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Executive  Summary 


This  report  details  work  conducted  under  the  auspices  of  the  DCIEM/ATG  Human  Factors  Study 
of  CC-130  Operations,  Phase  EL  The  contractor  was  part  of  the  Crew  Behaviour  Assessment 
Group  (CBAG)  in  Phase  1,  which  was  charged  with  developing  a  measurement  tool  for  assessing 
flightdeck  behaviours  indicative  of  safe  flight  behaviours.  This  final  report  first  outlines  the 
scope  of  the  contract  work  conducted  on  a  continuing  basis  in  support  of  the  Phase  1  objectives 
during  the  time  this  second  contract  was  in  place.  In  addition,  and  mainly,  this  contract  was  let 
in  order  to  further  one  of  the  recommendations  of  the  Phase  1  report,  namely  that  Air  Transport 
Group  (ATG)  consider  improvements  to  their  training  system  in  the  area  of  aircraft  systems 
knowledge.  This  final  report  therefore  summarises  the  goals,  nature  and  scope  of  that  work, 
includes  as  an  appendix  the  substantive  report  on  training  recommendations  (in  the  form  of  a 
fully  costed  proposal  for  training  improvements),  and  makes  further  observations  and 
recommendations  for  the  way  ahead  in  systems  training.  Information  sources,  influences,  and 
contacts  are  listed  to  aid  those  continuing  with  this  training  revision  effort. 
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Introduction:  The  CC-130  Human  Factors  Project 


The  current  contract  was  let  in  the  context  of  beginning  the  second  phase  of  the  CC-130  Human 
Factors  Project,  though  it  also  contributed  to  on-going  taskings  mandated  within  the  first  phase. 
In  order  to  summarise  the  history  of  the  project  to  date,  and  thereby  provide  a  context  for 
summarising  the  work  conducted  under  this  contract,  consider  this  quote  from  the  final  Project 
report  authored  by  Hendy  et  al.  (1998,  p.  ii): 

"In  January  1994,  the  Defence  and  Civil  Institute  of  Environmental  Medicine  (DC1EM)  was 
tasked  by  Air  Command  (AIRCOM)  to  study  human  factors  issues  in  CC-130  Hercules 
operations  within  what  was  then  Air  Transport  Group  (ATG)  —  now  a  component  of  1  Canadian 
Air  Division.  This  tasking  reflected  a  perception  that  the  accident  record  within  the  Canadian 
Forces  (CF)  fleet  was  significantly  worse  than  the  record  of  other  fleets  involved  in  similar 
operations.  In  response  to  this  tasking,  a  joint  DCIEM/ATG  team  was  fanned.  The  activities  of 
this  study  group  were  funded  by  the  Chief  of  Research  and  Development  (CRAD). 

The  first  activities  of  the  team  involved  a  review  of  relevant  literature,  a  survey  of  Squadron 
attitudes  regarding  the  priority  of  operations,  and  an  examination  of  CC-130  accident  boards. 
The  CC-130  Hercules  entered  service  with  Air  Transport  Group  in  1960.  Prior  to  1980  there 
had  been  two  A  Category  accidents  within  the  fleet.  Since  1980  there  have  been  six  major 
accidents  with  the  destruction  of  seven  a ircr aft  and  the  loss  of  44  lives.  Ho  common  cause 
factors  were  apparent  in  the  board  findings  that  would  suggest  that  the  same  set  of  deficiencies 
were  acting  in  these  accidents,  although  human  factors  issues  were  implicated  in  all. 

At  the  outset,  the  study  team  decided  to  take  an  approach  to  this  tasking  based  on  controlled 
experiment  and  field  observations.  Through  an  empirical  approach  it  was  hoped  to  present 
objective  data  that  would  define  the  extent  of  any  problems  identified,  and  provide  a  bench  mark 
against  which  various  interventions  (e.g.,  effects  of  experience  level,  fatigue,  training )  could  be 
assessed  with  regard  to  their  eventual  contribution  to  improved  safe  flight  performance.  To  this 
end,  the  team  assembled  two  working  groups  with  the  following  responsibilities: 

Crew  Behaviour  Assessment  Group  ( CBAG)  —  to  develop  and  validate  a  battery 
of  behaviourally-based  measures  that  could  be  used  to  distinguish  between  those 
behaviours  indicative  of  safe  flight  performance  and  those  that  are  not. 

Flight  Performance  Assessment  Group  (FPAG)  —  to  develop  and  validate  a 
battery  of  measures  based  on  aircraft  status  and  flight  path  that  could  be  used  to 
detect  departures  from  safe  flight  performance. 

There  have  been  two  Phases  to  this  study  to  date,  with  a  third  Phase  to  follow.  In  Phase  I 
(January  94  to  July  96)  the  two  working  groups  attempted  to  develop  and  validate  their 
measurement  batteries.  During  this  phase  a  number  of  observations  were  made  that  led  to 
certain  recommendations  being  made  to  ATG  for  changes  to  the  training  system.  In  Phase  II  the 
requirements  for  implementing  these  changes  were  explored  and  a  costed  proposal  was  put  to 
ATG  in  June  1997.  The  completion  of  the  Phase  II  activities  marked  the  end  of  the  study  group’s 
mandate  under  the  terms  of  the  original  tasking. " 
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I  Goals  of  the  Contract 


The  contract  goals  originally  centred  on  the  empirical  thrust  of  the  Phase  1  mandate:  to  develop  a 
set  of  observational  and  evaluative  metrics  of  human  flight  performance  that  could  serve  as  a  tool 
for  measuring  the  effects  of  various  factors,  and  later  interventions,  thought  to  relate  to  the  safety 
of  CC-130  flight  operations.  While  the  CBAG's  work  in  Phase  1  was  designed  as  but  a  first  step, 
a  measurement  development  and  validation  stage  for  the  tool,  the  controlled  series  of  full-motion 
simulator  (SIM)  sessions  that  were  conducted  and  analysed  led  to  some  results  that  ATG  found 
interesting  and  telling.  Among  these  was  an  apparent  lack  among  some  -  especially 
inexperienced  -  crews  of  basic  aircraft  systems  knowledge. 

The  belief  held  among  DCEEM  and  ATG  that  these  knowledge  deficits  should  be  ameliorated 
immediately ,  led  to  a  change  in  focus  of  the  contract  as  originally  conceived,  a  change  away 
from  empirical  investigations  using  the  human  performance  metrics  and  toward 
recommendations  for  implementing  training  solutions  that  would  redress  the  systems  knowledge 
problem. 

In  addition  to  these  foci  on: 

1)  completing  the  Phase  1  data  base  construction  and  furthering  analysis  of  it  in  order  to  more 
fully  develop  and  validate  a  crew  performance  metric;  as  well  as, 

2)  making  recommendations  for  improved  systems  knowledge  acquisition  training;  an 
additional  focus  was  on 

3)  being  available  as  a  general  SME  resource  for  consultation  with  various  principle  or 
supporting  investigators  working  on  the  Project's  other  aspects. 

I  now  review  the  scope  and  methods  of  work  for  each  of  these  elements,  listing  literatures, 
contacts,  and  resources  (typically  the  result  of  conference  attendance  and  networking)  for  each 
along  the  way. 
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Scope  and  Methods  of  Work 


Crew  Resource  Management 


Creation  of  A  Unique  Data  Base 

The  current  contract  began  during  the  latter  stages  of  the  first  phase  of  the  project,  with  an  intent 
to  examine  empirically  the  human  factors  thought  to  contribute  to  the  assumed  lowered  flight 
safety  standards  in  the  CC-130  fleet.  These  lowered  standards  were  presumed  to  have  led  to  the 
recent  poor  accident  record.  To  that  end,  the  initial  work  covered  under  the  contract  involved,  in 
part,  a  continued  focus  on  building  an  empirical  data  base  and  a  defensible  tool  for  assessing ’ 
human  flightdeck  performance  —  at  both  the  individual  and  team  levels  of  analysis.  This 
culminated  in  a  completed  database  from  the  Phase  1  validation  study  of  the  CB  AG.  It  will  be 
recalled  that  only  the  data  collection  was  covered  by  the  prior  contract.  Final  coding,  analysis, 
interpretation,  and  preparation  and  presentation  of  results  were  covered  by  the  present  contract. 


In  particular,  thanks  to  the  work  of  Dr.  Megan  Thompson,  who  largely  supervised  the  effort  on  a 
sub-contractual  basis,  the  complete  set  of  23  CC-130  crew  simulator  (SIM)  sessions  is  now 
available  as  a  resource  for  analysis  by  DCIEM  scientists.  Scientists  external  to  DCIEM  will  have 
to  wait  to  use  the  resource  until  the  ethically  required  step  of  securing  permission  from  the 
individuals  involved  to  have  their  sessions  viewed  and  analysed  is  finished.  This  permission  is 
necessitated  because  of  the  promise  of  confidentiality  made  during  the  study  to  flight  crew 
personnel.  The  confidentiality  assurances  were  needed  to  ensure  high  participation  rates  of 
crewmembers  and  to  reassure  them  that  their  participation  was  under  conditions  of  "no  jeopardy" 
to  their  careers.  (Indeed,  the  negative  reaction  and  objections  of  one  aircraft  commander  who 
inadvertently  discovered  that  ATG  personnel  had  judged  his  performance  as  recorded  on  the 
videotapes  bolsters  the  argument  that  gaining  such  permissions  is  crucial;  it  took  a  considerable 
effort  from  our  team  to  assuage  his  concerns). 

The  CBAG's  Phase  1  report  of  the  validation  stage  of  metric  development  (Banks  et  al.,  1997), 
completed  during  the  present  contract,  was  based  on  the  data  from  just  12  of  the  23  crews 
observed  under  controlled  SIM  conditions  —  those  six  crews  judged  most  proficient  and  those 
six  judged  least.  This  analysis  proved  complex  and  time-consuming.  It  involved  such  steps  as 
the  multiple  viewing  of  videotapes  leading  up  to  the  development  of  the  coding  scheme, 
articulation  of  categories,  multiple-judge  coding  of  the  tapes  for  reliability  analysis,  re-  ’ 
articulation  of  the  coding  categories,  performance  rating  made  by  three  independent  pilots 
invited  to  DCIEM  for  viewing  the  tapes,  selection  of  extreme  groups  for  construct  validation,  and 
so  forth.  Results  for  this  stage  are  reported  in  Thompson,  Jamieson,  and  Hendy  (1997). 

Under  the  purview  of  the  present  contract,  this  initial  work  was  further  extended.  First,  naive 
coders  were  trained  in  the  categorisation  scheme.  Because  of  the  degree  of  effort  required  to 
view  and  exhaustively  code  each  tape,  and  the  unavailability  of  suitably  expert  resources  to  be 
devoted  to  carry  out  this  task,  this  painstaking  stage  was  conducted  by  several  trained  observers 
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who  were  not  pilots  (some  supported  by  this  contract,  others  not,  for  example,  DCEEM  summer 
student  interns).  While  this  undoubtedly  added  to  the  time  taken  to  perform  the  coding,  it  also 
suggests  that  non-experts  may  be  employed  in  such  analyses  successfully  in  future  after  suitable 
familiarisation  and  training. 

To  complete  the  full  database  analyses,  all  23  crew  tapes  were  copied  (for  safekeeping  and 
security),  and  fully  transcribed  (a  step  not  undertaken  in  the  Phase  1  analysis  of  even  the  12  data 
points  retained).  One  set  of  tapes  was  time-stamped  for  later  temporal  analysis  (that  analysis, 
conducted  by  Dr.  Thompson,  and  reported  in  the  Phase  2  report  under  Thompson  (1998),  was  not 
covered  under  the  terms  of  this  contract;  the  results  have  also  been  submitted  for  publication, 
Thompson,  Jamieson,  &  Hendy,  1999).  Multiple  coders  then  coded  crew  behaviour  into  the 
behavioural  typologies  developed  (for  details  see  Banks  et  al.,  1997;  and  Thompson  et  al.,  1997; 
the  final  behavioural  metric  completed  under  the  contract  is  presented  in  Appendix  A). 

The  entire  process  took  hundreds  of  hours  of  sub-contractor  and  contractor  time  to  complete,  but 
eventuated  in  a  unique  resource  —  a  data  base  of  23  fully  transcribed  simulator  missions  run 
under  controlled  scientific,  rather  than  training,  conditions  in  an  event-based  scenario  (cf., 

Lauber  &  Foushee,  1981;  Prince,  Oser,  Salas,  &  Woodruff,  1993;  Seamster,  Hammond,  & 

Edens,  1995).  The  design  of  the  scenario,  geared  toward  creating  more  naturalistic  conditions  of 
decision  making,  captaincy,  and  crew  performance,  complete  with  inter-related  system  causality 
in  terms  of  aircraft  systems  failures,  serves  as  a  valuable  analytic  and  anecdotal  resource  for  CC- 
130  crew  performance  and  operations,  as  well  as  more  generally  for  team  performance  on  the 
flightdeck  of  multi-place  aircraft.  Its  future  further  analysis  holds  the  promise  of  revealing,  at 
least  on  a  descriptive  basis,  the  correlates  of  safe  and  proficient  flight  performance,  as 
exemplified  in  the  "best-performing"  crews  (i.e.,  as  independently,  and  "objectively"  judged  by 
check  airmen).  It  is  hoped  that  such  future  analysis  of  this  set  of  time-stamped,  transcribed,  and 
typologised  missions  will  yield  further  clues  as  to  the  individual  and  team  behavioural  profiles  of 
flightdeck  performance  excellence. 

Where  to  go  to  find  the  database:  Consult  with  Dr.  Megan  Thompson,  now  a  DS  at  DCIEM, 
under  whose  watchful  eyes  are  kept  the  physical  tapes,  as  well  as  the  transcriptions  and  coding 
masters  and  results. 


Validation  Study  Report,  Presentations,  and  Paper 

After  Dr.  Thompson  was  hired  by  DCIEM  as  a  DS  in  November  of  1996, 1  worked  with  her  in 
analysing  and  preparing  the  validation  study  report  that  was  included  as  part  of  the  Banks  et  al. 
(1996)  report  on  Phase  1.  Results  were  presented  as  part  of  a  daylong  presentation  at  DCIEM,  as 
well  as  a  report  to  the  Commander,  8  Wing.  Also  under  the  purview  of  the  contract  were  Dr. 
Thompson's  briefings  to  the  squadrons  about  the  results  of  the  study. 

In  addition,  another  version  of  the  Phase  1  results  was  communicated  at  the  International 
Symposium  on  Aviation  Psychology  in  April  of  1997.  I  contributed  to  both  the  abstract  that  was 
accepted  for  presentation  at  that  Symposium  (which  was  submitted  while  Dr.  Thompson  was  still 
part  of  the  contract  team),  as  well  as  the  subsequent  write-up,  on  which  Dr.  Thompson  led  the 
way  (Thompson,  Jamieson,  and  Hendy,  1997). 
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Keeping  Abreast  of  CRM  Issues 


Throughout  the  course  of  this  work,  Drs.  Thompson  and  Jamieson  kept  abreast  of  developing 
issues  in  CRM  both  in  general  and  at  ATG.  This  included  such  activities  as  reading  and 
incorporating  ideas  from  such  sources  as: 

•  the  literature  on  team  performance; 

•  the  net-based  CRM  discussion  group  and  developers  forum  (crm-devel@db.erau.eduk 

•  a  recent  volume  on  naturalistic  decision-making  (Zsambok  &  Klein,  1997). 

The  fruits  of  this  labour  and  thinking  is  to  be  found  in  the  revised  version  of  the  contractor's  final 
report  on  Phase  1  of  the  Project,  where  the  literature  review  on  CRM  issues  is  updated  from  the 
original  draft  report  of  March  1995  (see  Jamieson,  1999). 

In  addition  to  these  sources,  much  useful  information  was  garnered  at  the  International 
Symposium  on  Aviation  Psychology,  as  there  were  many  presentations  and  workshops  pertinent 
to  CRM  issues.  Conference  notes  from  Workshops  attended  are  attached  as  Appendix  B.  As 
one  example,  note  the  high  degree  of  integration  of  human  factors  teaching  in  all  aspects  of  flight 
training  as  presented  by  Wiggins  and  Henley  in  their  workshop  (materials  are  entitled: 
"Integrating  Human  Factors  into  a  Flight  Training  Syllabus").  In  this  workshop  they  argue  for 
the  need  to  teach  human  factors  throughout  training,  a  view  that  presages  the  CRM  course 
revision  submitted  by  Keith  Hendy  as  part  of  this  contract  work  (Hendy  &  Ho,  1998). 

Finally,  in  March  of  1997  the  contractor  attended,  with  a  large  contingent  from  DCIEM,  ATG's 
two-day  course  on  CRM  entitled  Aircrew  Co-ordination  Training  (ACT).  Notes  and 
observations  from  this  course  are  attached  as  Appendix  C.  Subsequent  to  this  course  attendance, 
the  contractor  focused  almost  exclusively  on  the  systems  knowledge  acquisition  aspect  of  the 
scope  of  work,  leaving  the  development  of  a  suggested  revision  to  ACT  training  to  the  contract's 
scientific  advisor,  Keith  Hendy.  While  keeping  abreast  of  all  written  materials,  my  contributions 
in  this  area  after  that  time  were  small.  Mr.  Hendy  and  Dr.  Thompson  took  over  that  part  of  the 
substantive  work,  with  Dr.  Thompson  focusing  on  issues  of  creating  a  proposal  to  demonstrate 
via  evaluative  research  the  efficacy  of  any  CRM  course  changes  proposed. 


Systems  Knowledge  Acquisition 


The  CBAG  study  findings,  as  alluded  to  above,  led  to  the  recommendation  that  certain 
modifications  to  the  training  system  be  made  right  away  (i.e.,  not  waiting  for  the  results  of 
further  empirical  investigations  using  the  Phase  1  human  metric  tool).  These  included  the 
directive  to  modernise  the  learning  processes  by  which  systems  knowledge  is  acquired.  This 
directive  served  as  the  main  focus  of  the  contract.  The  main  product  of  this  element  (Jamieson, 
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1998)  was  my  contribution  to  the  final  Study  report  —  Volume  4  of  Hendy  et  al.  (1998)  — 
which  is  attached  as  Appendix  E. 

Sources  of  Information 

The  Volume  on  systems  knowledge  acquisition  is  the  synthesis  of  many  information  sources. 
While  it  is  a  re-iteration  of  materials  contained  in  Jamieson  (1998),  it  is  useful  to  list  down 
several  of  these  many  sources: 

•  ATG  systems  knowledge  training  course  materials; 

•  ATG-provided  general  materials  on  CBT,  use  of  distance  based  education,  etc.; 

•  Hypher's  report  on  CC-130  training  modernisation; 

•  Conversations  with  ATG  training  personnel  at  various  levels; 

•  Conversations  with  Smart  Board  training  module  developers  at  Warner  Lambert; 

•  Conversation  with  CFB  Edmonton  staff  re:  the  AUP; 

•  Conversations  with  Flight  Safety  International  personnel; 

•  Literatures  on:  intelligent  tutors,  case-based  and  problem-based  teaching,  self-directed 
learning,  distance  education,  life-long  learning,  risk  assessment,  etc.; 

•  Conversations  with  individuals  met  at  the  Society  for  Applied  Learning  Technologies 
conference  on  interactive  multimedia; 

•  Attendance  at  the  Society  for  Applied  Learning  Technologies  1996  Interactive 
Multimedia  Conference; 

•  Attendance  at  the  Industrial  Technology  Research  Centre  (ITRC,  University  of  Waterloo) 
conference  on  on-line  learning. 

All  of  the  ATG-provided  materials  are  being  returned  to  DCIEM.  Photocopies  of  the  most 
relevant  articles  from  the  literature  reviews  are  also  being  submitted.  Neither  of  these  batches  of 
materials  is  included  in  the  Appendices  of  the  present  document. 

A  copy  of  the  SALT  conference  proceedings  is  provided,  and  my  notes  from  that  meeting  are 
found  filed  in  the  well-organised  binder  from  that  meeting.  A  summary  of  the  most  pertinent 
presentations  is  appended  as  an  Appendix  to  the  Volume  4  substantive  report  (Jamieson,  1998).. 

The  ITRC  Symposium  materials  and  my  notes  are  appended  as  Appendix  D. 

Below  is  a  sampling  of  particular  individuals  whose  contributions  to  my  thinking  were 
influential,  or  who  would  serve  as  valuable  contacts  for  anyone  continuing  this  work. 
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Contacts 


Name 


Affiliation 


Topic  Area 


Learnings 


Maj.  Dan  Duggan 

ATG 

Maj.  Rick  Lukes 

ATG 

Lt.  Col.  Peter  Thorpe- 
Levitt 

ATG 

Sgt.  Dave  Pollard 

ATG 

Maj.  Pierre  Naud 

ATG:  Chief  Technical 
Instructor 

Brad  Waddell 

Flight  Safety  Canada  / 
Flight  Safety 
International 

Martin  Smith 

Aviation  Human 

Science 

Richard  Pappas 

Lambert  Multimedia 
514-322-0630 

Robert  C.  Hughes 

Cubic  Applications 

Elias  Awad 

Exsys 

Alan  Blizzard 

McMaster  University 
Instructional 
Development  Office 

Overall  training 
philosophy/resources 


Systems  training 


Training  and 
development 


Hydraulics  instructor 


Adaptation  of  smart 
board  lessons  for 
pilots 


systems 


Situation  Awareness 


Smart  Board  system 
simulators 


Team  performance 
simulations;  after 
action  review  as 
training  aid 


Expert  systems 
software  for 
knowledge  extraction 


Case-based  teaching 


Distribution  of 
training 


CBI;  Smart  Boards; 
426  hardware 
infrastructure 


New  methods,  Hypher 
report,  etc. 


How  module 
designed,  integration 
with  in-class  methods 


Using  low/high 
fidelity  short-cuts  to 
Smart  Boards  for 
pilots;  Organisational 
resistance  to  DCIEM 
authored  "fixes" 


FSI  review  leading  to 
questioned  efficacy  of 
intelligent  tutoring 
systems 


Phenomenology  of 
situational  awareness 


SmartBoard  Software 
architecture,  potential 
upgrades,  costs 


Metacognitive 
development  in 
training 


Org.  knowledge  dump 
of  experts  on 
particular  systems 


Feasibility  of  this 
approach  to  ab  initio, 
AQP  trainin 
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General  Consulting  to  OPIs/ATG 

In  addition  to  the  focus  on  CRM  and  knowledge  training,  there  were  several  other  contributions 
to  the  CC-130  Study  made  by  the  contractor,  and  supported  by  the  contract,  that  leveraged  my 
involvement  in  the  project  from  Phase  1  and  contributed  to  my  subsequent  development  as  a 
SME  in  the  area.  These  included: 

•  availability  as  a  general  consulting  resource  on  matters  of  operational  logistics,  strategy 
and  experimental  design  for  OPIs  planning  studies  within  the  scope  of  the  Project;  and 

•  liaison  and  consultant  on  the  captaincy  study. 


I  played  a  minor  role,  with  Dr.  Thompson,  in  helping  to  design  and  position,  for  ATG 
acceptance,  a  study  of  fixed  and  formed  crews  to  be  led  by  Cpt.  Helen  Wright.  Maj.  John 
Latullipe  (8  Wing)  was  part  of  those  consultations.  However,  the  study  did  not  receive  approval 
to  proceed. 

In  addition,  I  also  played  a  very  minor  role  in  the  conceptualisation  of  a  study  of  aircrew  fatigue 
on  long-haul  flight  operations  to  Bosnia,  led  by  Michel  Paul  (Paul,  Pigeau,  &  Weinberg,  1998). 

I  had  somewhat  more  of  a  role  in  the  study  on  Captaincy  conducted  by  Ghee  Ho,  a  York 
university  graduate  student  in  psychology,  who  is  conducting  his  research  under  the  guidance  of 
Keith  Hendy.  The  study  asks  ATG  CC-130  Hercules  aircrew,  of  all  positions,  to  "fill  out"  a 
computer-assisted  personal  interview  (CAPI)  regarding  their  conceptions  of  good  versus  poor 
captaincy.  Here,  I  provided  some  conceptual  input  in  the  design  stage  of  the  Personal  Construct 
Theory  technique  of  eliciting  constructs,  as  well  as  some  advice  on  developing  dependent 
measures  (indicators,  actually,  since  this  was  a  correlational  study).  Mainly,  I  helped  provide 
some  liaison  with  ATG  personnel  at  8  Wing  to  get  the  study  going  (i.e.,  gaining  acceptance, 
establishing  essential  contact  personnel,  etc.),  accompanying  Ghee  on  a  trip  to  Trenton.  I  would 
recommend  an  investigation  of  recent  advances  in  categorical  data  analysis  and  graphical 
presentation  (as  contained,  for  example  in  SPSS  v9.0)  as  another  way  to  analyse  the  data  beyond 
those  analysis  methods  thus  far  proposed  to  be  used  in  the  study. 
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Future  Directions:  The  Way  Ahead  for  Training  CRM  and  Systems 
Knowledge 


It  is  my  belief  that  a  concerted  effort  needs  to  be  made  to  monitor  and  evaluate  the  state  of 
aircraft  systems  knowledge  and  CRM  within  the  Hercules  community  post-ground  school  and 
post-continuance  training.  In  addition,  this  needs  to  be  done  both  squadron-  and  fleet-wide,  and 
on  a  continuing  basis.  I  have  reviewed  in  Volume  4  of  the  final  Project  report  (Jamieson,  1998) 
the  need  and  opportunity  to  develop  pilot-targeted  case-based  lessons  and  scenarios  using  the 
Smart  Boards  as  a  basic  element,  and  then  distributing  these  to  the  squadrons  for  either  self-study 
or  facilitator- led  training.  This  training  distribution  to  optimise  the  conveyance  of  concepts  at 
426  and  skill  at  the  squadrons  in  the  realm  of  systems  knowledge  can  be  logically  applied  as  well 
to  CRM  training.  The  next  step  in  the  training  evolution  sequence  for  both  areas  is  nearly  at 
hand. 

After  the  publication  of  the  Volume  4  report  and  recommendations,  authors,  scholars,  and 
research  investigators  have  published  relevant  materials  on  the  topics  inherent  in  training  systems 
knowledge  (or  CRM  or  any  other  topic  for  that  matter).  I  have  submitted  (though  not  as 
Appendices  due  to  their  extensiveness)  two  additional  documents:  a  recent  (March  24,  1999) 
search  for  articles  on  simulation  in  training  that  may  be  valuable  to  those  carrying  forward  on 
these  training  recommendations,  and  a  current  (March  25,  1999)  listing  of  companies  providing 
aeronautic  services  —  software  and  hardware,  training,  etc. 

Beyond  these  most  recent  sources,  we  have  to  look  no  further  than  the  published  proceedings  of 
the  9th  Bi-Annual  International  Symposium  on  Aviation  Psychology  (all  references  are  1997)  to 
find  evidence  of  active  authorship  and  research  in  many  of  the  new  conceptual  teaching  and 
learning  areas  cited  in  Jamieson  (1998)  concerning: 

•  Piagetian  concepts  of  accommodation  and  assimilation  (Witiw) 

•  Principles  of  andragogy  (vs.  pedagogy)  and  engaged  life  long  learning  (Harris  &  Conrad) 

•  Self-directed  and  problem-based  learning  (Hill) 

•  Developing  self-awareness,  self-critiquing  and  metacognitive  skills  (Anderson  &  Henley) 

•  Aeronautical  decision-making  and  situation  assessment  using  PC-based  trainers  (Barber; 
Jentsch,  Irvin,  &  Bowers) 

•  PC-based  aviation  training  devices  (PCATDs)  (Emanuel;  Phillips  et  al.;  Talleur  et  al.;  Talyor 
et  al.) 

•  Balancing  ground  school  with  OTJ  (Mattoon) 

•  Use  of  videos  (Shaffer) 

This  year's  l(f  Bi-Annual  International  Symposium  (May  1999)  promises  new  developments  as 
well,  if  the  breadth  and  depth  of  topics  to  be  explored  as  indicated  in  advance  Symposium 
announcement  literature  and  workshop  line-ups  are  a  reliable  indication. 

Finally,  recent  advances  in  web-based  software  now  allow  for  the  running  of  simulation  groups 
over  the  internet.  In  this  paradigm,  flight  crews  can  be  constituted  from  distant  physical  locales. 
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and  can  be  brought  together  for  scenario  sessions  executed  in  real  time.  While  the  intent  would 
not  be  a  high  fidelity  representation  of  flight  systems  (though  that  too  will  be  possible  soon 
enough),  the  kinds  of  crew  captaincy  and  decision  making  skills,  crew  resource  management 
skills,  as  well  as  systems  knowledge  inter-relations  that  have  been  the  focus  of  the  contract  work 
can  be  tested.  Mid-  or  low-fidelity  event-based  scenarios  that  teach  higher  order  cognitive  skills 
and  task  and  teamwork  skills  can  be  mounted  with  relatively  few  technical  challenges. 

The  contractor  is  currently  helping  to  develop  a  group-at-a-distance  type  of  simulation  scenario 
to  be  deployed  on  a  web-site  now  under  construction  (the  site  is  currently  up  and  running  as 
'www.scenario.competitor.com';  it  is  worth  a  visit  to  see  some  state-of-the-art  functionality  in 
interactive  web  design).  Members  of  the  intended  group  will  be  brought  together  in  real  time 
and  asked  to  play,  as  a  group,  our  event-based  scenario  simulation.  Though  the  focus  is  on 
democratic  groups  rather  than  teamwork,  the  design  principles,  functionality  and  potential  are 
similar  to  what  might  be  rolled  out  for  the  Hercules  community.  Regular  role  plays  of  this  type, 
which  intermix  squadrons  in  the  constitution  of  their  crews,  will  allow  uniform  norms  and 
implicit  safe  flight  behavioural  standards  to  develop  more  quickly  fleet-wide.  And,  of  course, 
squadron-specific  operational  requirements  can  also  be  the  focus. 

The  work  is  being  conducted  as  part  of  a  consortium,  but  principle  contacts  of  value  to  this 
project  include: 

1.  Peter  Warrian,  President,  Environics  Technologies  Incorporated,  a  specialist  in  organisational 
knowledge  retention,  assessment  and  development;  and 

2.  Ken  K.  Campbell,  President,  Competitor  Communications  Incorporated,  a  web-design 
company  (Ken  is  a  regular  contributor  to  the  Star's  weekly  Thursday  section  on  emerging 
technologies.  Fast  Forward,  and  former  programmer  with  experience  in  military  simulation). 
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Conclusions 


Technological  advancements  in  training  delivery  software  have  now  reached  the  point  where 
self-directed,  problem-based  learning  can  be  considered  anew  as  a  primary  method  of  training 
aircraft  systems  knowledge  via  computer-based  training.  Sound  principles  of  andragogy  must 
provide  a  basis  for  the  construction  of  a  repertoire  of  training  cases  that  imparts  strong 
conceptual  knowledge  at  426  Squadron  and  continues  its  development  with  examples  to  be 
assimilated  during  OJT.  Such  distributed,  just-in-time  training,  with  a  phased  delivery  like  that 
envisioned,  will  put  extra  demands  upon,  but  also  further  reinforce  the  value  of,  staff  trainers. 
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Appendices 


For  the  purposes  of  minimising  reproduction  costs  and  volume,  the  following  materials  often 
summarise,  but  do  not  include,  the  materials  carried  back  from  conferences  and  meetings. 
However,  single  copies  of  these  materials  are  being  submitted  with  the  Contractor's  Final  Report. 


Appendix  A:  Revised  Crew  Behaviour  Scheme 

On  the  following  pages  is  a  description  of  the  final,  revised  crew  performance  metric  as 
developed  under  this  contract.  Recall  that  a  central  construct  was  the  shared  mental  model  and 
that  these  observable  metrics  focused  primarily  on  the  verbal  cockpit  communications  of  aircrew 
as  indexing  the  presence  of  these  (directly  unobservable)  mental  models. 


Mental  Model  Coding  Categories 


NOTE:  A  COMMUNICATION  MAY  BE  CODED  WITH  MORE  THAN  ONE  MENTAL  MODEL 
DOMAINS  AND  FUNCTION  Z  .  1  . 1  . ...  1 . ..  ''”'7 


CONTENT  DOMAIN 
OF 

COMMUNICATION 

CODE 

DESCRIPTION 

EXAMPLES 

MISSION 

M 

Statements,  questions,  commands,  etc. 
relating  to  the  time-limited,  medical 
purpose  of  the  flight 

“We  will  ERO  the  organ 
through  the  cargo  door.” 
(AC-medic)  “If  you  have  any 
requirements  or  requests  you 
can  relay  them  to  me  through 
the  loadmaster  once  we  are 
enroute.  Is  there  anything 
special  you  want  us  to  take 
care  of  right  now?” 

AIR  PICTURE 
(GEOGRAPHY) 

G 

Any  communication  (statement, 
question,  command,  etc.)  relating  to  the 
aircraft’s  physical  place  in  space, 
including  airways 

“Plan  on  24  left  to  Toronto.” 
“From  Trenton  to  Toronto  the 
VOR  is  36  miles.” 

SYSTEMS 

S 

Any  communication  relating  to  any  of 
the  mechanical,  electrical,  or 
navigational  systems  on  board  the 
aircraft.  That  is,  any  utterance  that 
refers  to  the  adjusting  of  a  system  will 
be  classified  as  system-related,  even  if 
that  system  is  geographical  in  nature 
(see  example). 

“For  departure  brief  right  seat 
let’s  start  with  Trenton  on  the 
TACAN  and  we’ll  check  the 
localizers  and  switch  them 
over  to  Campbellford  before 
we  start  rolling.” 

“Just  tell  me  when  you  want 
the  DME  on  Campbellford.” 

WEATHER 

w 

Any  communication  relating  to  weather, 
e.g.,  wind  direction,  gusts,  icing 
conditions. 

The  winds  are  going  to  keep 
coming  from  the  north.  Must 
be  a  front  moving  through.” 

PROCEDURES 

and 

CHECKLISTS 

CHLST 

Any  communication  referring  to 
checklists  actioned  or  to  be  actioned, 
specific  procedures,  also  any 
communications  referring  to  clearances 

“OK,  whenever  you  are 
ready,  you  should  probably 
do  a  checklist  here.” 

“OK  when  you  are  ready 
right  seat  we’ll  ask  them  for 
start  clearance.” 

Table  continues 
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Mental  Model  Coding  Categories  (cont’d.) 


FUNCTION  OR 
PURPOSE  OF 
COMMUNICATION 

CODE 

DEFINITION 

EXAMPLES 

AWARENESS 

AW 

Simple  declarative  statements 
concerning  the  present  states,  and 
immediately  impending*  changes 
in  the  states  of  systems,  the 
mission,  air  picture,  weather, 
checklists.  These  statements 
serve  to  maintain  and  update  the 
changing  mental  model  and 
short-term  goals. 

*as  distinguished  from 
preplanning 

“Levelling  6,000  in  the  turn.” 

“OK  8  miles  back  from  Campbellford  in  the 
TACAN.” 

“We’ve  got  a  hydraulic  low  pressure  light  on 
#2.” 

“OK,  I’m  going  to  push  the  power  up.” 

“Gear  is  up.” 

CROSS-CHECKING 

CC 

Explicit,  overt  noting  of 
deviations  with  respect  to  another 
crewmember's  actions,  planning, 
calculations  etc. 

‘You’re  off  the  glide  slope.” 

“Watch  your  airspeed.” 

“You  probably  want  to  keep  your  airspeed 
down.” 

“You’re  kind  of  diverging  off  the  west  from 
the  VOR ...” 

OK,  if  you  want  to  fight  her  back  down  to  4 
and  show  me  up,  that  would  be  good.” 

IMPLICATIONS 

IMP 

Explicit  and  overt  recognition  of 
the  consequences  or 
ramifications  of  present  system, 
mission,  weather,  checklists,  and 
geographical  states  and  the 
interactive  effects  of  these  states 
upon  each  other. 

“Just  considering  your  worst  case  scenario, 
would  you  be  interested  in  lowering  your 
gear  and  flaps  into  position  now,  just  in  case 
the  other  pump  goes?” 

“The  only  thing  I’ve  got  doubts  about  is  the 
antiskid  system  because  it  is  part  of  the 
touchdown  relay.” 

■”  other  systems  lost,  we  have  no  utility  so 
we  want  to  make  sure  that  the  brakes  go  to 
emergency.” 

PREPLANNING 

PP 

Explicit  statements  of  future 
intentions  or  plans  regarding 
systems,  the  mission,  air  picture, 
checklists  &  procedures 

“Okay,  so  we  are  15  miles  back  from 

Simcoe  and  if  we  do  a  teardrop  so  that  we’re 
back  on  the  047  ...  from  there  we  do  a 
procedure  to  get  us  into  24R.” 

“350  a  good  trot  to  Campbellford  till  we’re 
airborne  and  we’ll  pick  it  up  and  hold  it. 

257  out  of  Campbellford  to  TO.” 

“Depending  on  RWY  conditions,  I’d  go  to 
emergency  or  deactivate  your  antiskid  just  to 
be  safe  and  basically  be  gentle  on  your 
brakes.” 

Table  continues 
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Mental  Model  Coding  Categories  (cont’d.) 


ADDITIONAL  CODING 
CATEGORY: 

CODE 

DEFINITION 

EXAMPLES 

SYSTEM  KNOWLEDGE 

SK 

Spontaneous  &  explicit  statements 
by  crewmembers  regarding  how 
relevant  systems  operate,  indicating 
their  knowledge  of  systems  as 
distinguished  from  reading 
information  in  flight  manuals  (e.g., 
theZ-10). 

[re:  gear  override]  CO: 

“they’re  not  coming  up”  AC: 
“Hit  the  override  switch 
there.” 

TASK  PRIORITIZATION 

TSKPR 

Overt  &  explicit  direction  or  request 
for  direction  concerning  the  timing 
of  tasks  in  relation  to  one  another 

“Let’s  get  the  approach 
check  done  and  then  we’ll 
deal  with  that ...” 

“I’ll  finish  the  post-takeoff 
check  and  then  we’ll  deal 
with  the  landing  gear 
problem  before  we  get  too 
far  from  Trenton.” 

CREW  MONITORING 

CM 

Overt  &  explicit  statements  directed 
toward  assessing  and  addressing 
crewmembers’  workload, 
information  load,  or  vigilance 

“lust  let  me  know  how  you 
are  doing  over  there.” 

“How  are  you  doing  there 
CO?” 

“Everybody  got  the  ATIS?” 

OPEN-LOOP 

COMMUNICATION 

OLC 

Lack  of  response  of  any  kind  to 
directions,  suggestions,  queries;  also 
interruptions  that  interfere  with 
building  a  mental  model, 
information  transfer,  or  the 
development  of  a  team,  as 
distineuished  from  alerting 
interruptions  fe.g..  FE  must  break  in 
to  alert  the  AC  to  a  system 
emergency),  or  interruptions  that 
involve  one  crew  member  picking 
up  on  and  finishing  another’s 
thoughts  and  statements  (which  is 
evidence  of  a  shared  mental  model). 
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|  Appendix  B:  International  Symposium  on  Aviation  Psychology  (1997):  Workshop 


The  following  notes  were  taken  during  the  contractor's  attendance  at  the  Ninth  Annual 
Symposium  on  Aviation  Psychology,  held  April  27-May  1,  1997,  in  Columbus,  Ohio  (USA) 
For  further  information  on  these  workshops  and  sessions,  the  reader  is  referred  to  the 
symposium's  published  "Proceedings  of  the  9lh  International  Symposium  on  Aviation 
Psychology,"  R.S.  Jensen  &  L.A.  Rakovan  (Eds.),  Columbus,  OH:  The  Aviation  Psychology 
Laboratory,  Ohio  State  University. 


Workshop  14:  Facilitating  the  Integration  of  Human  Factors  Education  into  the  Operational 
ying  Curriculum,  Mark  Wiggins  and  Irene  Henley,  Department  of  Aviation  Studies,  University 
of  Western  Sydney,  Macarthur.  (Materials  are  titled:  "Integrating  Human  Factors  into  a  Flight 
Training  Syllabus  ).  5 

•  Asserts  that  human  factors  training  should  start  on  the  first  day  and  continue  on  everydav 

full  integration  J  J  ’ 

•  Don't  necessarily  have  to  use  HF  jargon  to  make  effective 

•  Signs  and  cues  of  HF  phenomenon  (concrete  examples,  cases  studies)  should  be  imparted 
along  with  appropriate  countermeasures 

•  Teach  to  recognise  HF  challenges  and  deal  with  them 

•  Emphasis  was  placed  on  the  importance  of  the  trainer's  role  in  imparting  understanding  of 

CBT  is  emphasised  as  a  valuable  learning  tool:  However,  it  is  asserted  that  effective  training 
especially  of  human  factors  issues,  still  needs  person-interaction  as  a  necessary  even 
essential,  element 

•  Training  the  trainers  becomes  especially  important  when  the  trainers  serves  as  models  for 

professionally  appropriate  attitudes  and  motivations 

•  Teach  and  model  explicit  self-evaluation  and  meta-cognitive  skills  so  as  to  promote  life-long 

•  Workshop  materials  included  in  final  contact  submission 


Workshop  5:  The  Aircrew  Behavioural  Compass:  A  Descriptive  Model  for  Categorising  and 
Understanding  the  Personal  Behavioural  Styles  of  Pilots,  Marvin  Karlins,  University  of  South 
Florida,  Singapore  Airlines. 


•  Primarily  a  personality  approach  to  flightdeck  behaviour 

•  e.g.,  halo  effect  of  the  nice  guy/good  atmosphere  pilot;  1st  officer  doesn't  want  to  destroy 
positive  atmosphere;  or,  e.g.2  presence  of  management  person  leads  to  non-challenging 
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•  why  right  seats  don't  challenge  left 

•  goal  to  increase  interpersonal  effectiveness  (though  this  left  undefined  and  unmeasured, 
presumably) 

•  Karlins  believes  personal  styles  can  be  trained  in  or  out  via  good  instruction/mentoring  and 
proceduralising  the  personality  out 

•  Good  instruction  includes  many  video  recreations 

•  All  briefs  therefore  very  important,  especially  tone  of  them 

•  Constant  practise  key  to  building  this  into  the  fabric  of  regular  ops 

•  Once  fleet  behavioural  styles  identified,  report  incidences  and  set  policy  to  change  behaviour 

•  My  assessment:  too  much  emphasis  on  the  personality  aspect;  workshop  itself  a  little  Mickey 
Mouse,  facile 

•  However,  suggests  we  should  apply  epistemic  analysis  (processes  of  freezing  and  unfreezing) 
to  the  total  crew  to  get  the  truest  picture  of  the  total  epistemic  environment 


Workshop  2:  Understanding  and  Preventing  Crew  Error,  Dr.  Patrick  Veillette,  Aviation  Safely 
Consultants. 


•  As  this  Workshop  was  booked  at  the  time  of  registration,  I  pursued  the  author  and  arranged 
to  have  course/workshop  notes  sent  later.  Note  that  the  numbering  is  not  always  consecutive 
and  may  reflect  an  edited  version 

•  My  impression  is  that  this  is  a  comprehensive  laundry  list  of  human  factors  implicated  in 
instances  of  unsafe  cockpit  behaviour 

•  Also  provided  are  recommendations  for  what  to  teach  in  a  CRM  course,  though  the  list  is 
similar  to  what  is  currently  provided  in  many  existing  courses. 


Note:  Workshop  materials  for  all  three  workshops  are  included  with  the  final  contract  submission 


Appendix  C:  ATG  Aircrew  Coordination  Training  (ACT)  Notes 


The  following  notes  were  taken  during  the  contractor's  attendance  at  the  CF  Air  Command 
Aircrew  Coordination  Training  (ACT)  Session  conducted  March  10-March  1 1,  1997  by  Air 
Transport  Group  trainer  Maj.  Frank  Pelltier,  8  Wing,  CFB  Trenton. 

These  notes  do  not  concentrate  on  the  content  of  what  was  delivered  in  the  course,  so  much  as 
the  nature  of  the  andragogy  employed.  Suggestions  and  opportunities  for  improvement  are 
noted,  but  reflect  the  author's  own  approach  to  training.  Ordering  of  notes  follows, 
approximately,  the  order  in  which  topics  are  treated  in  the  course. 


Does  ACT  Work? 

•  Presentation  of  "data"  on  the  efficacy  of  ACT 

•  CAE-Link  Corporation  workbook  used  many  places,  with  reports  of  increased  error 
protection  in  the  months  following  the  course  of  8-10% 

•  Organisational  evaluations  attribute  from  28%  to  8 1%  reduction  in  accident  rates  to  this 
course  or  other  ACT  courses 

•  Clearly,  hard  to  make  the  attribution  directly  to  the  course  in  the  absence  of  formal 

evaluations  studies  (with,  for  example,  wait  list  controls);  evaluation  confounded  with  a  host 
of  other  factors 

•  The  key  hypothesis  is  still  that  changing  attitudes  works,  that  it  spreads,  that  organisational 
culture  then  supports  this,  and  that  improvements  are  attributable  to  the  training  of  new 
attitudes  (all  testable  hypotheses,  but  none  tested  so  far) 


ACT  Instructional  Delivery:  Main  Methods  _  _ 

•  Course  materials  used  such  as  icebreakers,  videos,  exercises;  still  mostly  chalk  and  talk  from 
instructor  —  need  more  and  better  aids/materials 

•  Course  book  positioned  as  a  generic  building  block  to  be  revisited  again  and  again  and  then 
customised,  but  few  concrete  recommendations  are  given  as  to  how  to  do  this;  tough  to  teach 
common  denominators  in  such  limited  time  given  heterogeneity  of  backgrounds  of  attendees 

•  Content  delivered  via  MS  Power  Point 

•  Method  of  delivery  often  war  stories:  share  experience  (confidential,  no  risk,  supportive 
environment  promoted,  but  is  it  even  close  to  being  achieved?) 


ACT  Course  Atmosphere:  Intention  vs.  Reality 

•  Goal  to  create  a  no-risk,  involving  environment 

•  Get  the  sense  people  are  still  not  really  comfortable  with  discussing  this  openly  after  a  couple 
of  days  (perhaps  inhibition  from  presence  of  large  contingent  from  DCIEM) 
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Course  Goals 

•  Overall  goal  of  course:  impart  changed  knowledge,  attitudes,  skills;  while  first  two  may  be 
achievable  to  some  (evaluable,  but  unevaluated)  degree,  tough  to  do  the  last 


CC-130  Context  for  Course 

•  Review  of  CC-130  accidents 

•  Past  mission  analysis  rife  with  operation  of  hindsight  bias  (my  opinion) 

•  Attribution  of  half  of  causes  of  crashes  to  ACT  (48%  to  lack  of  aircrew  co-ordination;  22% 
loss  of  Situational  Awareness;  19%  violation  of  regulations;  11%  poor  decision  making) 

•  Also  cite  FedEx  CRM  course  training  and  evaluation  where  3%  got  "worse"  (by  what 
measurement  criteria?);  harps  a  bit  (missionary-like?)  on  "good  attitude" 


HF  and  CRM:  What  it  is? 

•  Listing  of  CRM  phenomenon  done  by:  describing  it,  defining  it,  giving  examples,  trying  to 
define  when  it  is  most  likely  to  occur,  how  to  avoid  (e.g.,  SLOJ,  Strength  of  an  Idea,  Copilot 
Syndrome,  Halo  Effects,  Excessive  Professional  Courtesy,  Hazardous  Attitudes,  Hidden 
Agendas,  etc.);  divides  into  2  rough  classes:  cognitive/mindset  and  social  interaction 

•  Assessment  of  this  strategy:  elements  are  there  but  not  enough  meaningful  integration  to 
allow  long  term  assimilation  and  integration  of  learning 


Countermeasures  .  .  . . .  .... 

•  Tools  for  combating  these  phenomena:  aircrew  co-ordination  cycle,  SOPs,  AESOP  Model, 
Two-Challenge  Rule,  Most  Conservative  Response  Rule,  Assertive  Statements,  Shock 
Statements,  "Am  I  Safe"  Checklist,  etc. 

•  Teach  meta-awareness  about  when  such  tools  are  used  and  should  be  appropriately  used; 
combine  with  Awareness-Implications-Planning  (AIP)  Model  (Hendy). 

•  Teach  more  specific  skills  for  these;  e.g.,  "Am  I  Safe"  lists  what  to  check,  but  not  how  to 
check  them,  what  criteria  to  judge  against,  all  the  concrete  aspects 


Accident  Review 

•  Accident  case  studies  presented  to  build  base  of  knowledge  via  post-hoc  causal  analysis 
(little  evidence  such  exercises  may  transfer) 


Policy  and  Regulations 

•  Section's  goal  is  to  describe  their  role,  why  they  have  come  about,  analyse  the  causes  of 
deviations  from  P&R,  list  inadvertent  and  purposeful  departures  from 

•  Assessment:  still  a  lot  of  laundry  list  teaching 

•  Something  interesting  in  terms  of  teaching  could  be  done  with  purposeful  departures  in  the 
sub-class  called  'necessary  departures':  should  plan  and  get  feedback  from  crew  on  proposed 
options,  make  a  clearly  communicated  conscious  decision  (to  build  the  mental  model  that  is 
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shared),  plan  carefully  (limit  setting,  contingency  pre-planning),  directly  brief  (corroborating 
mental  model  allows  others  to  contribute  timely  information,  expertise).  Suggest  they  do  a 
case  study,  then  a  role-play  of  how  to  do  this  well. 

Personality  Styles 

•  AC  Personality  Types:  interesting  but  perhaps  better  taught  as  a  situational  than  personality 
set  of  constructs/phenomena:  state  vs.  trait  issue 

Leadership 

•  Next,  sections  on  Power  Bases,  Leadership,  Creating  an  Assertive  Statement,  Destructive 

Patterns,  Constructive/Corrective  Patterns  . . 

•  Again,  verbal  review  only,  almost  totally  conceptual  with  few  concrete  examples 

•  On  Leadership:  task  priority  by  people  skills  interaction  conceptualisation;  say  what  to  do 
(namely,  build  confidence,  trust,  and  support  and  gauge  the  reserves  and  capabilities  of  crew 
members)  but  not  how  to  do  it 

•  Creating  an  Assertive  Statement:  say  how  to  do  it  (steps:  opening,  state  concern,  state 
problem,  offer  solution,  gain  acceptance)  but  despite  how  easy  it  would  be  to  provide 
behaviour-based  training  for  this,  NONE  is  done 

•  Note:  by  this  point  in  the  day  we  are  behind  schedule  and  it  is  hot  in  the  room;  topics  are 
skimmed  over  in  the  most  cursory  way  and  nobody  is  taking  notes  or  being  responsive  any 
longer  (instructor  has  lost  them  for  the  day) 

•  People  skills  like:  giving  expression  to  positive  and  negative  feedback  is  trainable,  but  is  left 
untrained,  just  talked  about  generally  and  in  a  cursory  way  —  no  detail,  no  concrete 
examples  or  case  studies;  refer  to  voluminous  literature  on  feedback  and  how  to  give  it  in 
organisational  behaviour  texts 

Aircraft  Commander's  Responsibilities  __ 

•  AC's  Responsibilities:  very  important  topic  from  point  of  view  of  cross-training  roles  and 

responsibilities;  other  airmen  must  understand  constraints/expectations  that  ACs  work  under 
m  order  to  provide  good  followership;  shared  mental  model  must  have  these  as  primary 
component  J 

•  Late  in  day  and  we're  behind  so  again,  this  most  important  topic  skimmed  over 

•  Again,  the  what  is  imparted,  but  not  'how'  on  topics  like:  establishing  clear  authority  and 
leadership,  adjusting  use  of  authority  to  situations,  creating  an  atmosphere  where  crew 
participation  can  flourish,  pre-briefing  cockpit  responsibilities,  and  quick  resolution  of 
disruptions 

Aircrew's  Responsibilities 

•  Respect/support  AC  authority,  assertiveness  with  respect,  when  in  doubt  speak  out,  giving 
best  information/judgement  to  AC  in  a  timely  way,  etc. 

•  Again,  these  are  the  very  micro-behavioural  elements,  paralinguistics,  body  language,  etc 
that  can  be  practised  and  trained  successfully 
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Aircrew  Communications 

•  Again,  much  of  this  material  is  new  and  useful  for  airmen,  but  is  glossed  over  due  to  the 
pace;  unlikely  to  be  remembered  because  it  is  not  experienced  nor  practised;  low 
assimilation  to  schemas  so  long-term  integration  and  retention  in  doubt 

•  The  most  behavioural  of  the  elements  covered  so  far,  so  most  amenable  to  training 

•  Needs  to  have  much  better  case  study  materials  and  time  for  practice  practice  practice 

•  Example  given  about  use  of  verbal  information  to  describe  an  ambiguous  stimulus;  to  what 
effect?  (Point  of  the  example  was  lost  on  me) 

•  Senders  and  receiver  responsibilities:  too  much  lecturing 


Workload  and  Performance  Under  Workload 

•  Reviewed  are:  Causes  and  effects  of  high  and  low  workload,  how  to  recognise  (cues), 
manage 

•  Inadequate  treatment 


Available  Resources 

•  Review  of  internal  and  external  resources 

•  Demonstration  of  using  other  members  as  resources:  fun  but  of  questionable 
teaching/training  value 

•  Conclusions  something  like:  Identify  resources,  discuss  and  decide  how  to  use  them,  use  the 
AESOP  model;  elements  are  there  but  need  more  clarity  and  cohesion 


Situational  Awareness 

•  Again,  review  of  concept,  symptoms,  how  to  recognise,  countermeasures  to  avoid  losing, 
technique  for  regaining 

•  Return  to  idea  of  knowledge,  attitudes  and  skills  as  training  preparation  and  first  defence,  but 
with  little  time  or  ability  to  train  skills 

Overall  Assessment 

•  Either:  1.  Concretise  the  training  and  give  much  more  time  to  skill  development  (which  may 
entail  a  longer  course),  or  2.  Concentrate  on  strong  conceptual  knowledge  of  human  factors, 
decision  making  and  aviation  psychology  in  general,  to  provide  sound  schematic  base  for 
later  OTJ  learning  of  ACT  skills  (apportion  skills  to  OJT). 

•  Either  tack  involves  the  needed  development  of  behaviour-based  teaching  methods  for  skill 
development  of  CRM  and  communications  skills 

•  Little  content/development  of  fundamental  left  seat  requirements  like  decision  making  under 
normal  and  anomalous  flight  conditions  and  cockpit  resource  management  skills  and 
requirements 

•  This  is  the  first  and  best  opportunity  to  build  a  community-wide,  shared  and  institutionalised 
mental  model  of  safe  flight  operations  as  practised  by  a  multi-place  interacting  crew 
members 
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Appendix  D:  ITRC  Symposium  Materials  and  Notes 


The  following  notes  were  taken  during  the  contractor's  attendance  at  the  Information  Technology 
Research  Centre  (ITRC)  Conference  entitled,  "Online  Learning  1997:  Effective  and  Available". 
ITRC  is  located  at  the  Davis  Centre,  the  University  of  Waterloo  (Tel:  (519)  888-4540). 
Associates  of  ITRC  who  helped  organise  and  fund  the  meeting  included  the  Educational 
Network  of  Ontario,  watAGE  Inc.,  and  SMART  Technologies.  The  conference  was  held 
December  16  and  17  ,  1996,  and  the  contractor  filled  in  for  Dr.  Thompson  on  the  conference's 
second  day,  as  she  was  unable  to  attend.  This  conference  examined  current  methods  for  on-line 
learning,  including: 

•  Creating  and  maintaining  materials; 

•  Communicating  among  instructors  and  the  learning  audience; 

•  Distributing,  assessing,  and  using  data; 

•  Usability  testing  and  analysis;  and 

•  Integrating  online  learning  into  education,  government,  business  and  agency  operations. 

The  reasons  for  the  contractor’s  presence  at  this  conference  were  four-fold: 

•  To  learn  more  about  electronic  support  for  conventional  learning  methods; 

•  To  learn  more  about  the  development  and  support  of  multimedia,  online  learning  via  the 
Web 

•  To  learn  more  about  Smart  Board  technologies  and  other  recent  advances  in  technology 
designed  for  instructional  delivery 

•  To  evaluate  the  viability  of  a  web-based  training  solution  for  the  CC-130  Hercules 
community. 

The  conference  outline  is  attached.  On  day  2,  key  addresses  germane  to  the  contractor's  °oals 
were  given  by:  ° 

•  Kevin  Hooper,  Smart  Technologies  (smarttech.com)  and  Toni  DiPetta,  Educational 
Network  of  Ontario,  on  "Smart  Communications:  Visual  Human  Interchanges  in  Real 
Time"; 

•  Paul  Beam,  watAGE  Inc.,  "The  Online  Learner:  Developing  Audience  Expectations"; 

•  Mark  Chignell,  Faculty  of  Engineering,  University  of  Toronto,  "Make  It  Work  For  Me- 
The  Efficiency  of  Adapting  Systems  to  Users";  and 

•  Robert  Moore,  InContext  Corp.  (www.incontext.com),  "Information  Organised  for 
Human  Exchange".  (Substitute  speaker  for  absentee  presenter;  not  in  conference  agenda 
nor  on  attendance  list) 
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Notes: 


Hooper  and  DiPetta: 

•  Distance  based  learning  linking  up  to  64  points  via  LAN,  WAN,  Internet,  etc. 

•  Remote  clients  need  software 

•  HTML  databases;  graphics-based:  create  objects  and  link  them  (e.g.,  test,  graphics,  sound, 
clipart,  ppt,  HTML,  etc.) 

•  Enables  CUCME,  video-conferences 

•  Graphic  links  to  homepage 

•  Smartboards  allow  collaborative  notes  to  be  taken,  displayed  to  all  users,  saved  to  file 

•  Creates  an  audit  trail 

•  My  assessment:  Smart  Technologies  provides  powerful  group-at-a-distance  learning  tool  to 
facilitate  communication  and  instruction 


Beam: 

•  On-line  instruction/training  requires  necessary  triad  of  structured  data  (content), 
administrative  systems,  communications  tools 

•  Key  is  in  instantiating  self-directed  learning  principles 

•  Provide  progress  feedback  to  learners  in  several  ways:  email,  webchat,  newsgroups, 
comments  from  evaluators,  useraccounts,  projects,  instructor's  comments 

•  Learners  should,  in  parallel,  evaluate  the  method  of  delivery  using  several  methods  (as 
above)  to  iterate  delivery  efficiency 

•  Create  safe  environment/coach  learners  to:  query  re:  their  problems 

•  Classify  these  problems 

•  Record  audit  trail  for  students  of  various  abilities 

•  My  assessment:  classification  of  problems  gives  strong  clues  as  to  the  development  of  mental 
models  and  where  they  are  under-developed 

•  Analysis  of  systematic  differences  between  C  and  A  students  in  the  audit  trail  should  allow 
the  development  of  metacognition  models  amenable  for  use  by  the  C  students:  they  should  be 
able  to  leam-question-check  and  engage  in  other  learning  processes  more  like  those  students 
who  are  more  successfully  apprehending  the  knowledge  when  they  are  shown  these  more 
successful  strategies,  and  then  model  them,  incorporate  them  and  practice  them. 


Chignell: 

•  Needs-based  approach 

•  Give  away  the  pedagogy/andragogy  —  make  transparent 

•  Make  Learners  into  teachers/authors 

•  Teach  by  having  learners  publish  (case  studies,  projects,  etc.);  builds  on  the  old  adage  that  to 
really  understand  something  you  have  to  be  able  to  teach  it 

•  Give  them  all  the  tools  needed  to  be  researchers/authors 
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beadaptable611  ***  t0°1S  Sh°Uld  be  general  PurPose  rather  than  specific,  but  they  should 

My  Assessment:  Have  incident  and  accident  databanks  in  the  Hercules  community  and 
unrecorded  personal  incidents  (or  bogus  ones),  written  into  case  studies  for  instructional 
delivery  (via  smartboard  or  other  method)  by  the  pilots  themselves  as  part  of  their  training 
requirement.  Their  analysis  of  the  cases  and  events  should  be  detailed  and  extensive  it 
should  be  evaluable,  and  should  incorporate  feedback  as  received  by  instructors  initially  and 
as  received  by  users  as  the  case  is  used  and  critiqued  * 

This  strategy  helps  with  the  problem  of  insufficient  case  content  and  builds  the  critical  mass 
of  cases  for  course  delivery  quickly  without  relying  solely  on  already  over-burdened  trainers- 
also  captures  organisational  knowledge  of  senior  pilots;  allows  them  to  articulate/displav 
their  superior  systems  and  flight  deck  management  knowledge;  their  cases  serve  as  fodder  for 
evaluation  of  many  aspects  of  safe  flight  performance. 


Moore: 

•  (Works  with  military) 

•  new  definition  of  "document"  as  "Information  Organised  for  Human  Exchange" 

•  electronic,  distributed,  compound  (multiple  contributors) 

•  a  frame  or  filter  for  objects/methods  that  act  on  the  data 

•  documents  are  not  just  data,  but  become  active  processes 

•  re-use  of  knowledge  important;  subdivide  learning  components  into  small  chunks  for  re-use 
again  and  again 

•  documents  provide  components 

•  the  way  the  components  are  linked  (e.g.,  HTML)  are  important 

•  take  components  from  context  to  context 

•  SGML  authoring  (supports  all  HTML) 

•  SGML  database-schema  +  data  dictionary  +  data 

•  SGML  as  object-schema  language  —  new  and  evolving 

•  Document  =  object  frame;  programming  =  authoring 

•  from  HUI  to  GUI  to  DUI  (Document  User  Interface) 

•  my  assessment:  have  loose  categories  to  start  with,  mark  up  only  the  text  they  use  (echo 
technique),  record  users  choices  as  important  data;  build  the  mental  model 
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Appendix  E:  Human  Factors  of  CC-1 30  Operations:  Volume  4,  Training  Systems 
Knowledge 


The  following  report  was  written  as  the  main  substantive  deliverable  of  the  systems  knowledge 
acquisition  portion  of  the  contract.  It  was  included  as  Volume  4  of  Hendy  et  al.  (1998),  and  is 

reprinted  here  with  page  numbering  as  a  stand-alone  document. 

Its  official  citation  is: 

Jamieson,  D.  W.  (1998).  Human  factors  of  CC-130  operations.  Volume  4:  Training  systems 
knowledge  (DCIEM  Report  98-R-17).  Toronto,  ON:  Defence  and  Civil  Institute  of 
Environmental  Medicine. 
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EXECUTIVE  SUMMARY 


tasmarLTATT'"?  T?  “  ,he  June  1996  briefin*  ,0  11,6  Commander,  Air 
1  ransport  Group  (ATG),  served  as  the  impetus  to  the  work. 

•  Raise  the  overall  level  of  systems  knowledge  among  ATG  aircrew  by  developing  teaching 
aids  and  courses  to  accelerate  the  acquisition  of  knowledge  and  to  compensate  for  lowered 
fleet  experience  levels  through  advances  in  training  program  delivery. 

In  this  Volume,  systems  knowledge  training  is  first  placed  within  a  general  risk  management 
scheme  as  an  overall  causal  framework  for  securing  safer  and  more  effective  operations  within 
the  CC-130  community  of  1  Canadian  Air  Division  (1  CAD).  Systems  knowledge  acquisition  is 

thrieve^of  th  "?k.  r’  °peratmg  both  as  a  precondition/psychological  precursor  of  risk  at 
the  level  of  the  individual,  and  as  a  systemic  line  management  deficiency  at  the  level  of  the 

organisation  Formal  evaluation  of  the  state  of  systems  knowledge  fleet-wide  is  needed  to 
provide  feedback  to  the  organisation  in  its  attempts  to  manage  latent  risks. 

Current  systems  knowledge  training  practices  within  the  organisation  are  reviewed  next 
including  past  assessments  of  the  need  for  modernisation  of  training.  It  is  concluded  that  while 
current  training  practices  are  sound,  thorough,  and  professional  in  execution,  there  remains  room 
for  improvement  because  downsizing  and  experience  loss  have  impeded  the  ability  of  8  Wing 
Trenton,  to  achieve  training  modernisation  .  &■> 

A  selected  review  of  best  training  concepts  and  practices  is  presented,  based  on  the  recent 
i  erature  on  education  and  training.  Those  concepts  best  suited  to  the  CC-130  training 
environment  are  identified.  The  key  areas  covered  include:  g 

•  the  importance  of  life-long  learning; 

•  a  just-in-time  knowledge-to-skill  training  model; 

•  problem-based  (i.e.,  case-based),  self-directed  learning; 

effective  use  of  interactive  multi-media  for  training  at  the  Operational  Training  Unit  (OTU) 
for  on-the-job  training  (OJT)  and  self-study;  1 

methods  for  the  development  of  superior  mental  models  of  aircraft  systems  and  their 
operational  interdependencies;  and 

the  need  for  evaluation  at  three  levels:  trainee  self-evaluation,  OTU/OJT  formal  student 

evaluation,  and  organisation- wide  monitoring  and  evaluation. 

A  set  of  recommendations  is  then  presented,  for  vaiue-added  training  system  improvements  in 
the  area  of  aircraft  systems  knowledge  acquisition.  These  recommendations  are  based  on  the 

inchidtag  traimnS  PraCtkeS  tha'  C,m  bS  delivercd  within  CUITeM  organisational  restraints. 


•  distributing  more  formal,  detailed  skill  training  to  the  Squadrons,  while  keeping  conceptual 
mental  model  building  the  focus  of  the  OTU; 

•  teaching  the  Awareness  Implications  and  Planning  (AIP)  model  as  a  conceptual  approach  at 
an  early  stage  and  throughout  training  so  that  it  becomes  habitual, 

•  using  more  case-based  teaching,  supported  by  video  simulations  of  good  as  well  as  poor 
systems  handling,  making  sure  that  aircraft  system  interdependencies,  and  the  need  for 
implicational  thinking  and  pre-planning  to  deal  with  these,  are  illustrated; 

•  forging  partnerships  in  the  world-wide  CC-130  community  to  build  a  library  of  cases 

quickly; 

•  leveraging  the  Smart  Board  software  pieces  —  deploy  the  Smart  Boards  at  both  the  OTU  and 
OJT  within  the  context  of  both  problem-based  learning  and  self-study,  combining  them  with 
a  case-based  method,  and  intelligent  tutoring  for  self-directed,  continuous  performance 
improvement  —  build  the  tutor  to  provide  feedback  that  will  develop  pilots’  abilities  for 
objective,  independent  self-evaluation  of  the  state  of  their  systems’  knowledge  (i.e.,  meta¬ 
knowledge)  and  motivate  self-initiated  learning; 

•  develop  appropriate  standards  and  evaluation  tools  for  this  type  of  training, 

•  conduct  formal  evaluation  studies  to  document  the  efficacy  of  these  value-added 
improvements;  and 

•  create  a  non-jeopardy  database  to  monitor  and  track  fleet-wide  the  development  of  superior 
systems  knowledge. 

The  Volume  concludes  with  a  summary  of  the  deliverables,  the  implications  for  implementation 

that  are  discussed  throughout,  and  the  resource  requirements  implied  by  adoption  of  these 

recommendations. 
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INTRODUCTION 


This  is  Volume  4  of  a  six  volume  series  describing  the  outcome  of  a  human  factors  study  into 
CC-130  operations  (see  References  [1-5]  for  details  of  the  remaining  documents  in  this  series). 
In  this  Volume  the  issue  of  needed  improvements  to  the  training  of  systems  knowledge  is 
addressed.  After  reviewing  the  reasons  why  this  was  identified  as  an  area  needing  improvement, 
such  training  is  placed  within  an  overall  risk  management  framework  for  securing  safer  and  more 
effective  operations  within  the  CC-130  community.  Current  systems  knowledge  training 
practices  within  the  organisation  are  then  reviewed,  including  past  assessments  of  the  need  for 
modernisation  of  training.  Following  this,  a  selection  of  best  training  and  education  concepts  is 
reported,  based  on  the  recent  literature  on  education  and  training,  showing  which  of  these 
concepts  are  best  suited  to  the  CC-130  training  environment.  This  leads  naturally  to  a  set  of 
recommendations  for  value-added  training  system  improvements  in  the  area  of  aircraft  systems 
knowledge  acquisition.  It  is  argued  that  there  is  a  need  to  quantify,  with  suitable  empirical 
documentation,  the  magnitude  of  these  training  improvements  after  deployment,  and  to  monitor, 
fleet-wide,  the  evolving  norms  regarding  pilots’  systems  knowledge.  This  would  provide 
feedback  to  the  organisation  in  its  attempts  to  manage  latent  risks.  Implications  for 
implementation  of  these  ideas  for  both  the  Operational  Training  Unit  (OTU)  and  for  On-The-Job 
Training  (OJT)  are  drawn  out  and  reported  as  to  complete  the  analysis.  Finally,  the  resource 
requirements  implied  by  such  changes  are  enumerated. 

The  following  recommendation  made  in  the  June  1996  briefing  to  the  Commander,  Air  Transport 
Group  (ATG),  served  as  the  impetus  to  the  work. 

•  Raise  the  overall  level  of  systems  knowledge  among  ATG  aircrew  by  developing  teaching 
aids  and  courses  to  accelerate  the  acquisition  of  knowledge  and  to  compensate  for  lowered 
fleet  experience  levels  through  advances  in  training  program  delivery. 

This  recommendation  resulted  from  the  observation  of  inadequate  in-depth  aircraft  systems 
knowledge  demonstrated  by  several  pilots  in  the  Crew  Behaviour  Assessment  Group  (CBAG) 
behavioural  measurement  validation  study  [6].  These  pilots  had  difficulty  recognising, 
diagnosing,  or  ameliorating  some  of  the  aircraft  systems  failures  incorporated  into  this  event- 
based  scenario.  While  a  few  limited  observations  do  not  make  a  fleet-wide  trend,  these 
observations  were  embraced  by  ATG  as  perhaps  typical  of  a  more  general  problem  —  at  least  on 
an  intuitive  and  anecdotal  level  since  adequate  measures  of  systems  knowledge  proficiency  do 
not  currently  exist  that  would  document  fleet-wide  trends  in  this  type  of  expertise. 
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SYSTEMS  KNOWLEDGE  IN  THE  MANAGEMENT  OF  RISK 


ithin  Reason  s  [7]  systems  approach  to  risk  management,  adopted  as  a  framework  for  the 
activities  of  the  CC-130  study  team  (see  Volume  1  of  this  series  for  a  more  detailed  discussion) 
the  acquisition  of  aircraft  systems  knowledge  can  be  seen  as  occurring  relatively  early  in  the 
chain  of  organisation-wide  activities  in  which  1  Canadian  Air  Division  (1  CAD)  engages.  A  lack 
of  adequate  and  appropriate  knowledge  of  aircraft  systems  can  therefore  be  conceptualised  most 
generally  as  a  latent,  as  opposed  to  an  active,  failure  in  the  accident  sequence.  Such  lack  of 
knowledge  can  be  classed  as  occurring  within  two  of  the  risk  management  system  states  earlier 
e  ined,  namely  Preconditions’  and  ‘Line  Management  Deficiencies’.  These  two  states 
correspond  to  control  loops  3  and  4,  respectively,  of  Volume  1,  Figure  4  (repeated  below  as 
Figure  1,  for  convenience). 


FAILURE  TYPES 


FAILURE  TOKENS 


Figure  1.  Feedback  loops  and  indicators  for  the  management  of  system  safety 
(after  Ref.  [7],  Figure  7.9). 
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As  a  Precondition/Psychological  Precursor  of  Risk 

Inadequate  systems  knowledge,  possibly  caused  through  a  variety  of  factors  such  as  experience 
loss,  or  an  erosion  of  training  or  standards,  sets  a  precondition  for  failure  because  key  cockpit 
decision  makers  (Aircraft  Commanders  or  ACs)  are  inadequately  skilled  to  meet,  with  sufficient 
alacrity,  common  aircraft  systems  challenges,  outages  and  failures.  Unsafe  acts  become  more 
probable  in  a  context  of  lowered  or  inadequate  systems  knowledge,  and  the  more  unsafe  acts 
attributable  to  this  source,  the  more  likely  this  latent  failure  may  eventuate  in  accidents  when  1 
CAD’s  defences-in-depth  are  repeatedly  tested  due  to  this  cause. 


It  is  obvious  to  aircrew  (anecdotal  evidence;  see  also  the  study  of  dimensions  of  captaincy  in 
Volume  5  of  this  report)  that  a  key  dimension  of  leadership  and  captaincy  in  terms  of  both  safety 
and  production,  is  a  high  degree  of  systems  knowledge  proficiency.  No  matter  how  it  is  initially 
imparted,  or  later  developed,  embellished,  and  translated  into  skill,  systems  knowledge  is  a 
necessary  pre-condition  for  the  prevention  of  unsafe  acts,  and  for  the  recovery  from  them  as  well. 
In  the  terms  used  previously  [6,  8],  pilots  must  have  a  well-articulated  and  organised  mental 
model  of  each  aircraft  system  and  of  each  system’s  interactions  with  other  systems. 

These  mental  models  are  complex  cognitive  representations  of  the  various  systems’  functions. 
Such  cognitive  representations  allow  heightened  abilities  for: 

•  perception  and  situational  awareness; 

•  judgement; 

•  implicational/inferential  thinking; 

•  pre-planning;  and 

•  dynamic  decision  making. 

Each  of  these  functions  of  aircraft  systems  mental  models  allows  for  faster  and  more  accurate 
assessment  and  action,  buying  valuable  time  in  the  Information  Processing  (IP)  model’  trade-off 
between  the  time  required  to  do  what  must  be  done  and  that  time  available  to  do  it  [9].  In 
addition,  superior  knowledge  representation  of  systems  implications  allows  for  preplanning  and 
decision  making  that  can  ‘buy  time’  when  necessary,  again  insuring  that  the  IP  Model  ratio  does 
not  exceed  one.  Under  such  conditions  of  superior  aircraft  systems  proficiency,  advanced 
systems  knowledge  representations  become  skill-like  in  their  implementation.  This  is  a  simple 
description  of  the  preparedness  that  comes  with  high  skill  levels;  well-developed  mental  models 
about  aircraft  systems  contribute  to  more  effective  and  efficient  information  processing  about 
those  systems,  as  well  as  to  a  reduction  in  workload  stress  and  an  improvement  in  both  safe  and 
efficient  flight  performance  in  real-time  operations.  Simply  put,  this  state  of  mental  model 
functioning  can  be  taken  as  a  definition  of  expertise  with  aircraft  systems. 


1  The  IP  model  was  one  of  the  theoretical  positions  that  guided  this  study  (see  this  report,  Volumes  1  and  5).  The 
IP  model  claims  that  performance,  error  production  and  perceptions  of  workload  are  all  driven  by  the  time  pressure 
(i.e.,  the  ratio  of  time  required  to  time  available). 
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As  a  Line  Management  Deficiency  Risk 

The  development  of  superior  systems  knowledge  that  will  ensure  both  safe  and  efficient  flight 
performance  can  be,  and  is,  imparted  at  many  levels,  and  at  many  time  points  in  the  overall 
training  of  CC-130  pilots.  One  obvious  place  for  intervention,  as  well  as  for  monitoring  systems 
knowledge  to  provide  feedback  to  the  risk  control  system,  is  at  426  Squadron  —  the  OTU.  In  the 
context  of  lowered  fleet-wide  experience  levels  (see  Volume  1  of  this  report),  more  effective  and 
efficient  delivery  of  systems  knowledge  training  can  serve  to  compensate,  at  least  in  some 
measure,  for  a  loss  of  expertise.  It  was  a  goal  of  the  CC-130  study  team  to  identify  those 
teaching  philosophies,  approaches,  methods,  materials,  delivery  systems  etc.,  that  can  best  add 
value  to  an  already  redoubtable  training  effort  at  426.  In  leveraging  even  further  the  many  solid 
approaches  to  training  as  well  as  training  technologies  currently  in  place  at  the  OTU,  it  is  hoped 
to  be  able  to  recommend  improvements  to  training  delivery  that  will  push  trainees  more  quickly 
up  the  learning  curve  toward  high  proficiency.  It  is  also  hoped  to  be  able  to  document  the 
efficacy  of  these  improvements  with  suitable  empirical  evidence  regarding  their  effects,  in  the 
form  of  formal  evaluation  studies.  In  addition  to  demonstrating  the  efficacy  of  the  proposed 
improvements,  these  studies  will  serve  as  a  model  for  the  kind  of  evaluation  studies  needed  by 
the  organisation  to  track  the  state  of  systems  knowledge  ,  as  pilots  exit  the  various  OTU  courses. 

In  recognising  the  limited  exposure  that  pilots  have  to  426  Squadron,  components  of  training 
outside  the  OTU,  that  are  in  need  of  amplification  or  introduction,  are  also  foreseen.  In 
particular,  OJT  at  Squadron  is  also  a  focus  of  the  study  team’s  recommendations.  The 
distribution  of  superior  training  techniques  to  all  levels  and  kinds  of  training  exposure  is  the  best 
way,  it  is  considered,  to  achieve  a  fleet-wide  improvement  in  systems  knowledge.  As  in  the  case 
of  the  flight  school,  adequate  evaluation,  standards,  and  monitoring  must  be  put  in  place  for  OJT 
that  will  allow  multiple-source  feedback  on  the  normative  state  of  system  knowledge 
performance  fleet-wide.  This  feedback  will  inform  the  risk  management  system  at  a  deep  level 
(corresponding  to  control  loop  4  in  Figure  1)  of  potential,  or  latent,  failure  in  the  area  of 
knowledge  of  aircraft  systems.  Such  a  monitoring  system  is  particularly  important  and  timely 
given  the  near-term  upgrade  to  the  semi-glass  cockpit  for  the  CC-130,  because  past  research  has 
shown  that  systems  interdependencies  in  such  aircraft  exacerbate  the  difficulty  of  training  [10]. 
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REVIEW  OF  CURRENTTRAINING 


In  Phase  I  of  the  CC-130  study,  anecdotal  evidence  of  inadequate  aircraft  systems  knowledge 
among  some  crew  participants  from  the  crew  performance  validation  study  was  observed,  noted 
and  reported  to  ATG  [6].  In  Phase  n,  activities  were  directed  toward  identifying  and 
recommending  mechanisms  for  the  accelerated  acquisition  of  systems  knowledge  both  within 
formal  and  on-the-job  training.  Because  it  was  Aircraft  Commanders  (ACs)  who  exhibited  the 
clearest  instances  of  lack  of  systems  knowledge  in  the  CB  AG  study  (though  it  is  recognised  that 
it  is  likely  only  a  very  small  minority  of  ACs  that  is  not  currently  meeting  appropriate  standards), 
that  focus  was  on  the  acquisition  of  systems  knowledge  among  ACs  at  various  levels  of  training. 
The  current  Phase  II  activities  were  therefore  directed  toward: 

•  conducting  a  review  of  current  426  training  system  methods,  materials,  and  practices; 

•  conducting  a  selected  review  of  the  education  and  training  literature  —  both  theory  and 

current  practices;  and  J 

•  making  recommendations  for  value-added  improvements  to  the  training  system  in  terms  of 
effectiveness,  efficiency,  and  evaluation. 


Review  of  426  Squadron  Training 

The  review  of  426  Squadron  training  philosophy  and  regimes  included  interviews  with  trainers 
and  course  developers,  reviews  of  written  course  materials  and  limited  attendance  at  some 
courses.  The  focus  was  on  conversion  training  and  the  upgrade  course,  since  it  was  pilots’ 
(particularly  ACs’)  system  knowledge  more  than  systems  knowledge  at  any  other  crew  position 
that  most  directly  affected  the  outcomes  of  the  flights  we  observed  in  the  validation  study. 

This  review  confirmed  that  some  excellent  training  resources  exist  at  the  Squadron  currently  for 
the  acquisition  of  systems  knowledge,  and  that  a  starting  infrastructure  exists  to  expand  upon 
these  already-excellent  offerings.  It  is  considered  that  current  426  Squadron  training  is  well- 
grounded  in  education/training  theory,  is  both  comprehensive  and  thorough  in  coverage,  and  is 
professional  in  execution.  Future  efforts  at  adding  value  to  this  system  should  use  these  existing 
elements  as  a  base  for  what  would  be  characterised  as  a  small  to  moderate  re-engineering  of 
current  instruction.  However,  there  are  areas  that  can  be  leveraged  further,  some  that  would  be 
classed  in  the  area  of  training  effectiveness,  and  some  in  the  area  of  training  efficiency. 

Particular  strengths  observed  in  the  training  of  systems  knowledge  include: 

•  clear,  lesson-specific  instructional  goals; 

•  comprehensive  course  materials; 

•  effective  examples  and  scenarios; 


7 


•  use  of  a  multi-tiered  approach  (i.e.,  lectures,  mock-ups,  part-task  simulations,  full-motion 
simulations,  etc.); 

•  effective  use  of  cases; 

•  presence  of  experts  in  the  classroom  to  address  problems/questions;  and 

•  use  of  smart  board  technology  as  the  mechanism  of  instructional  delivery  in  certain  instances 
(i.e.,  fuel,  and  hydraulics  at  present). 

Areas  of  potential  improvement  at  426  Squadron,  and  for  OJT,  that  were  identified  include. 

•  striking  a  new  relative  balance  between  the  conceptual  content  and  the  detail  communicated 
about  systems; 

•  use  of  OJT/Squadron  Standards  Officers  to  continue  the  conversion  of  knowledge  to  skill  in  a 
way  consistent  with  and  co-ordinated  by  trainers; 

•  an  operations-specific  training  needs  assessment  conducted  at  the  various  Squadrons  to  tailor 
some  training  toward  customers  differing  in  role  requirements; 

•  updating  materials  and  examples; 

•  use  of  even  more  facilitation-type  teaching  with  case  study  examples; 

•  explicit  methods  for  developing  superior  mental  models  of  aircraft  systems  that  include 
awareness,  implicational  thinking,  planning,  and  decision  making  regarding  those  systems; 

•  explicit  methods  for  developing  mental  models  of  aircraft  system  interactions; 

•  opportunities  for  training,  developing  and  exhibiting  systems  knowledge  and  systems 
knowledge-relevant  decision  making  in  dynamic,  more  realistic,  event-based  scenarios  (low- 
to  high-fidelity); 

•  completion  of  the  outstanding  smart  boards  for  systems  not  yet  represented  and  taught  in  this 
mode; 

•  increased  functionality  of  the  smart  boards  (e.g.,  to  allow  and  stimulate  implicational 
thinking  about  systems); 

•  improved,  formal  evaluation  and  organisational  feedback  at  every  stage  of  training;  and 

•  provision  of  tools  and  motivation  for  career-long  self-study,  student  feedback  and  the 
development  of  self-evaluation  skills. 

The  review  of  systems  knowledge  training  was  done  in  conjunction  with  a  review  of  current 
education  and  training  practices.  The  latter  served  as  a  filter  through  which  to  view  current  CC- 
130  training  practices.  Before  turning  to  this  review,  it  is  instructive  to  compare  the  CC-130 
study  group’s  independent  assessment  of  systems  knowledge  training  circa  1997,  with  one 
conducted  by  ATG  in  1990.  In  some  respects  the  review  and  recommendations  made  some  7 
years  ago  strongly  foreshadow  the  position  taken  in  this  Volume. 


Hypher’s  Study  of  CC-130  Pilot  Training  Modernisation 

In  1983,  Mr.  Peter  Hypher,  then  Operational  Research  Advisor,  reviewed  the  CC-130  training 
systems  at  the  Operational  Training  Unit  as  part  of  the  Commander  ATG’s  initiative  to 
modernise  the  pilot  and  aircrew  training  systems  [11].  He  updated  his  review  in  1990,  under 
contract  to  the  Operational  Research  and  Analysis  Establishment  [12].  Both  reviews 
distinguished  between  the  concepts  of  cognitive  and  psycho-motor  skills  acquisition,  emphasised 
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the  need  for  a  progressive  learning  approach,  and  sought  to  encourage  the  adoption  of 
progressive  training  aids. 

Hypher’s  main  arguments  were: 

•  that  learning  of  cognitive  and  psychomotor  skills  within  ATG  took  place  largely  in  isolation 
and  needed  to  be  integrated/altemated  all  along  the  continuum  from  the  OTU  Conversion  to 
OJT,  to  Upgrade;  and 

•  that  the  ideal  training  system  would  incorporate  training  aids  ranging  along  a  continuum  from 
textbooks  and  lectures,  to  computer-based  training  (CBT)  lessons,  to  part-task  trainers  (PTT) 

to  cockpit  procedures  trainers  (CPT),  to  the  full  motion  simulator  (Sim),  and  finally  to  the 
aircraft. 


By  1990,  he  saw  CBT  as  desirable  and  feasible  and  spent  much  of  his  report  reviewing  and 
recommending  it  as  the  best  method  for  transmitting  basic  knowledge  and  training. 

Since  1990,  several  aids  have  been  added  to  the  training  system,  including  the  cockpit  mock-up 
(CMU)  arising  out  of  the  CC-130  Avionics  Update  Programme  (AUP),  and  the  Smart  Board 
systems  packages  (fuel  and  hydraulics  delivered,  with  remaining  systems  on  order:  electrics, 
propellers,  etc.),  that  go  a  long  way  toward  meeting  the  recommendations  Hypher  made. 
Importantly,  many  of  Hypher’s  observations  with  regard  to  the  needs  of  the  training  system  hold 
as  true  today  as  they  did  7  years  ago.  Organisational  stresses  in  the  form  of  downsizing  and 

fleet-wide  pilot  experience  loss  have  prevented  426  Squadron’s  ability  to  modernise  its  training 
as  rapidly  as  hoped.  6 

When  Hypher’s  arguments  are  applied  to  the  issue  of  systems  knowledge  acquisition  many  of 
the  assertions  and  recommendations  are  accepted,  but  issue  is  taken  with  several  points  too 
Without  exhaustively  reviewing  his  main  contentions  (which  stand  in  his  own  companion 
document),  the  areas  of  basic  agreement  with  his  training  system  vision,  and  areas  where  there 
are  differences,  are  summarised  below. 

It  is  agreed  with  Hypher  [12]: 

•  that  the  present  OTU  systems  knowledge  treatment  "...over-teaches  the  technical  detail  " 

(P-1'7); 

•  that  there  needs  to  be  more  emphasis  on  "...good  airmanship  and  airspace  awareness  initially 
than  on  technical  detail..."  (p.  22)  at  the  OTU; 

•  that  the  OTU  tends  to  teach  "...ten  years  of  systems  failures..."  (p.  3)  in  a  few  days  at  the 
Conversion  course  and  that  this  overloads  the  student; 

•  that  there  is  a  need  for  a  progressive  learning  approach  in  the  area  of  systems  knowledge 
acquisition  (p.  3); 

•  that  there  is  a  need  for  progressive  training  aids  to  teach  systems  knowledge  (p.  19); 

•  that  the  present  training  system  consists  (still)  of  too  much  chalk-and-talk  with’ regard  to 

systems  knowledge  (p.  7;  though  this  is  changing  with  the  introduction  of  each  new  Smart 
Board  module);  and 

•  that  a  rating  system  be  introduced  to  assess  the  training  value  for  each  aircraft  system  of  each 
operational  training  flight  (p.  77). 
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His  call  for  the  introduction  of  self-paced  study  is  also  accepted,  though  there  are  differences  on 
the  best  mechanism  for  this.  Hypher  recommended  traditional  CBT,  which  was  state  of  the  art  at 
the  time,  and  this  represents  the  main  area  of  apparent  discrepancy  between  the  recommendations 
of  the  CC-130  study  team  and  those  of  Mr.  Hypher.  While  it  is  true  that  traditional  CBT  has 
many  advantages  over  lectures,  including  shortening  systems  knowledge  acquisition  times  and 
bolstering  knowledge  retention,  the  developmental  efforts  and  costs  attached,  estimated  as  taking 
anywhere  between  250  to  800  person  hours  of  preparation  per  hour  of  CBT  instruction,  and  as 
costing  between  $15,000  to  $30,000  per  hour  of  final  CBT  product,  are  prohibitive.  Moreover, 
these  sorts  of  estimates  are  for  classical  CBT,  which  amounts  in  many  cases  to  expensive  page- 
turning  software  of  largely  text-based  material.  To  fully  realise  the  advantages  of  sound,  video, 
animation,  and  simulations  within  a  classical  CBT  environment  would  add  incrementally  to  these 
costs. 

ATG  has  not  in  the  past  chosen  the  classical  CBT  option  for  training  systems  knowledge,  and  in 
hindsight  the  study  team  concurs.  CBT  is  not  the  appropriate  alternative  even  at  this  time, 
especially  in  light  of  the  development  of  another  (but  relatively  less)  cost-intensive  vehicle  for 
instruction  —  the  Smart  Board  CDs.  These  can  powerfully  substitute  for  CBT  —  indeed,  they 
are  a  form  of  it  —  while  at  the  same  time,  it  is  believed,  achieve  considerable  training  efficacy  (it 
is  argued  below  that  this  training  efficacy  should  be  documented  empirically  via  formal 
evaluation  research  using  objective  performance  measures,  evidence  that  will  quantify  the  Smart 
Boards’  effectiveness  and  increase  their  marketability  in  other  C-130  communities). 

Recent  advances  in  authoring  systems  for  multi-media  training  suggest  that  using  commercially 
available  authoring  software  may  now  be  viable  for  the  creation,  maintenance  and  revision  of 
multi-media  courseware  for  the  CC-130.  This  flexibility  in  design  and  the  ability  to  amend 
lesson  plans  has  been,  heretofore,  largely  lacking  among  CBT  packages.  These  can  now  be 
usefully  and  easily  employed  as  courseware  generators  among  multi-media  literate  trainers.  It  is 
recommended  that  caseware  and  courseware  be  built,  placing  as  their  centrepieces,  the  Smart 
Board  system  simulator  module.  Emphasis  on  preplanning  and  decision  making  will  be 
manifested  as  lesson  objectives  that  surround,  and  are  placed  ‘on  top’  of  the  basic  Smart  Board 
capability.  Were  Mr.  Hypher  to  have  been  writing  today,  given  his  stated  recommendation  to 
integrate  sound,  video,  and  simulation  within  classical  CBT,  it  is  suspected  that  he  would  in 
agreement  as  to  the  training  potential  inherent  in  the  Smart  Board  systems. 

Another  key  area  of  disagreement  in  the  learning  model  adapted  for  the  current  review  of  CC- 
130  training,  as  compared  with  Hypher’s  own  recommendations,  is  his  statement  that  conversion 
training  should  emphasise  “...’how  to  operate  systems’  rather  than  ‘how  they  operate’...”  (p. 
129).  It  is  believed  that  a  sound  conceptual  basis  —  understanding  how  and  why  the  system 
operates  as  it  does  —  provides  a  key  knowledge  structure  within  which  technical  detail  and  skill 
can  be  incorporated  and  retained.  Enough  how  to  operational  knowledge  must  be  imparted  to 
allow  aircrew  to  contribute  as  useful  team  members  upon  exiting  the  conversion  course,  but  a 
sound  conceptual  understanding  is  a  necessary  pre-condition  for  differentiated,  integrated,  and 
retrievable  systems  knowledge.  This  is  especially  the  case  as  the  issue  (increasing  in  importance 
with  the  Avionics  Update  Project  [AUP])  of  interconnected  systems  functioning  [13]  is 
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broached.  Situational  awareness,  implicational  thinking,  pre-planning  ability,  and  decision 
making  are  all  facilitated  by  a  sound  mental  representation  of  systems’  causal  interactions. 

One  of  Hypher’s  stated  reasons  for  suggesting  CBT,  was  to  secure  self-study  opportunities  and 
access  for  students.  Digital  compression  and  other  computer  technology  advances  since  the  time 
of  his  writing  now  make  network-hosted,  fully  interactive  multi-media  CBT  with  simulation 
components  realisable  in  the  near  term  as  a  tool  for  systems  knowledge  training.  The  Smart 
Boards  represent  software  pieces  that  can  be  fairly  easily  adapted  to  be  self-paced,  self-directed 
distance  learning  applications,  concurring  with  and  delivering  Hypher’s  earlier  vision  for  career- 
long  currency  maintenance  and  performance  improvement. 


Review  of  Education  and  Training  Literature 

Several  of  the  areas  for  improvement,  identified  above,  were  inspired  as  a  result  of  a  selected 
review  of  the  current  literature  on  education  and  training.  Those  trends,  themes  and  principles 
judged  most  pertinent  to  CC-130  pilot  training  are  summarised  below  as  six  main  points.  At 
each  point  the  principle  involved  is  reviewed,  what  is  believed  to  be  its  best  application  within 
the  CC-130  training  environment  is  described,  and  the  implications  that  adoption  or  further 
extension  of  the  training  approach/principle  would  have  are  outlined.  Recommendations  are 
summarised  only  after  describing  all  the  points  in  order  to  avoid  redundancy,  because  many  of 
the  themes  are  strongly  inter-related.  Resources  that  may  prove  useful  in  future  implementation 
are  referenced,  and  listed  in  the  resource  appendix  attached  (Appendix  1). 

There  are  six  key  themes  involved  in  this  review.  These  range  from  motivational  factors  that 
will  ensure  more  effective  training  and  an  emphasis  on  conceptual  models,  all  the  way  to  specific 
ideas  concerning  appropriate  self-paced  multi-media  tools  that  will  simultaneously  improve 
training  efficiency,  effectiveness,  and  evaluation.  The  six  themes  to  be  described  are. 

1 .  The  Importance  of  Life-Long  Learning. 

2.  A  Just-in-Time  Knowledge-to-Skill  Training  Model. 

3.  Problem-Based  (i.e.,  Case-Based),  Self-Directed  Learning. 

4.  Effective  Use  of  Interactive  Multi-Media  for  OTU/OJT  and  Self-Study. 

5.  Methods  for  the  Development  of  Superior  Mental  Models  of  Aircraft  Systems  and  their 
Operational  Interdependencies. 

6.  The  Need  for  Evaluation  at  Three  Levels:  Trainee  Self-Evaluation,  OTU/OJT  Formal 
Student  Evaluation,  and  Organisation-Wide  Monitoring  and  Evaluation. 


The  Importance  of  Life-Long  Learning 

Life-long  learning  is  a  concept,  of  great  currency  value  of  late,  in  educational  theory  and  practice 
[14,  15].  Educators  and  trainers  have  long  recognised  that  the  pace  of  technological  invention 
and  change  means  that  the  half-life  of  most  knowledge  is  limited.  In  an  age  of  rapid 
advancement,  it  becomes  requisite  that  practitioners  in  every  field  keep  constantly  in  tune  with 
the  latest  trends  and  developments  in  their  areas  of  expertise.  Responsibility  for  the  pursuit  of 
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continuous  education  and  performance  improvement  throughout  the  career-  and  life-span  now 
vests  with  individuals  as  well  as  the  training  arms  of  organisations.  Life-long  learning  is  the  way 
of  the  future  for  everyone,  and  its  need  is  already  here. 


The  concept  of  life-long  learning  is  viewed  as  being  essentially  a  motivational  one.  Eventually, 
through  the  development  of  assessment  instruments  that  will  allow  the  organisation  to  carefully 
identify  screen,  and  select  those  candidate  pilots  in  possession  of  both  a  natural  propensity 
toward  self-improvement  via  self-motivated  study  and  a  natural  heightened  recognition  of  the 
need  to  stay  current,  1  CAD  will  be  able  to  select  individuals  who  come  bearing  the  right,  life¬ 
long  learning’,  stuff.  However,  such  diagnostic  measures  of  personal  motivation  and  selection 
are  not  yet  in  existence,  and  so  steps  that  are  systemic  and  situational  in  nature  must  be  put  in 
place  that  will  motivate  aircrew  toward  self-improvement  at  the  same  time  as  giving  them  the 
kind  of  tools  and  affording  them  the  kind  of  learning  environment  where  motivated  self-study 
can  flourish  early  on  —  and  be  maintained  throughout  the  career-span.  In  other  words,  1  CAD 
must  create,  even  more  than  it  has  already  largely  successfully  done,  the  means,  motive  and 
opportunity  for  life-long  learning. 


There  are  larger  issues  of  organisational  structure,  career  paths,  incentive  systems,  organisational 
commitment  and  the  like  that  will  need  to  be  broached  to  ensure  the  success  of  mstillmg  a 
philosophy  of  life-long  learning  [16].  This  Volume  will  focus  on  tools  for  self-directed, 
continuous  improvement  rather  than  on  these  larger  organisational  issues.  It  is  believed  that  the 
kind  of  balanced  training  model  proposed  (see  next  section)  will  make  training  more  enjoyable, 
(even)  more  respected,  and  more  motivating  than  it  is  currently,  in  keeping  with  the  kind  of  role 
(and  status)  within  the  organisation  that  the  training  system  must  occupy. 


Implications.  Occupational  environments  that  promote  life-long  learning  must  support  such 
learning  with  proper  tools  and  aiding.  Limited  teaching  budgets  suggest  that  continual 
improvement  and  education  is  done  most  cost-effectively  if  the  student  assumes  major 
responsibility  for  their  upgrading  via  self-study.  In  the  area  of  systems  knowledge  acquisition, 
this  suggests  that  appropriately  gauged  self-study  packages  be  available  to  pilots  on  an  on-going 
basis  This  also  implies  that  experts  be  available  for  consultation  in  support  of  self-study,  an 
that  software,  for  example,  be  available  to  provide  an  appropriate  degree  of  ‘help  when  needed 
in  the  absence  of  such  experts.  Help  can  range  from  the  provision  of  indexes  and  glossanes 
during  self-study  all  the  way  to  intelligent  tutoring  systems  (see  below).  Since  most  ot  these 
self-study  materials  are  likely  to  be  computer-based  (i.e.,  software),  this  also  implies  some 
investment  in  pilots’  computer  literacy  early  on  in  their  training  to  maximise  their  self-study  and 

life-long  learning  potentials. 


A  .Tust-in-Time  Knowledge-to-Skill  Training  Model 

Effective  training  starts  with  the  conveyance  of  knowledge,  by  which .  it  is  meant  that  a 
conceptual  understanding  of  the  subject  based  on  theory  and  examples  is  laid  down,  and  through 
practice  this  knowledge  becomes  appropriately  skill-based.  Simple  principles  of  knowle  ge 
acquisition  suggest  that  a  strong  conceptual  grounding,  or  infrastructure,  is  necessary  in  e 
knowledge  acquisition  process.  Later  detailed  understandings  are  best  learned  and  retained  when 


they  can  be  assimilated  to  an  existing  conceptual  ‘scaffolding’  (i.e.,  assimilation  of  knowledge  is 
always  easier  than  accommodation). 

In  the  present  instance,  this  conceptual  scaffolding  can  be  understood  in  terms  of  solid,  cognitive 
schemata,  or  mental  models,  of  aircraft  system  functioning.  These  mental  models  promote 
higher  order  cognitive  processing,  such  as  inference,  reasoning,  trouble-shooting,  etc.,  because 
they  represent  not  just  what  a  system  does,  but  how  and  why.  Once  a  strong  conceptual  mental 
model  is  in  place,  the  assimilation  of  details  in  practice  is  an  efficient  process.  It  is  considered 
that  some  training  benefits  may  accrue  if  the  limited  exposures  pilots  have  to  426  Squadron  is 
even  more  concentrated  than  at  present  on  improving  this  conceptual  understanding  (i.e.,  laying 
the  basis  for  superior  cognitive  representations),  while  OJT  continues  to  concentrate’ on  the 
assimilation  of  details  and  the  development  of  skill.  Over-teaching  of  detail  at  the  OTU  is  not  a 
wise  use  of  limited  ground  school  time. 

The  latter  point  relates  to  the  important  issue  of  the  timing  of  learning.  Learning  is  maximised 
when  concepts  are  acquired  ‘just-in-time’,  that  is,  when  they  are  actually  needed  in  practice; 
knowledge  that  is  transmitted  but  is  unrehearsed  quickly  atrophies  (as  anyone  who  has  taken  a 
course  but  not  applied  it  for  several  months  will  attest).  ‘Just-in-time’  is  a  term  that  strongly 
implies  skill  transmission  at  the  squadron  operational  level.  Therefore,  careful  consideration 
must  be  given  to  articulating  when  it  is  that  different  kinds  of  knowledge  are  optimally  acquired 
within  the  entire  training  system.  An  over-emphasis  on  detail  to  the  detriment  of  conceptual 
understanding  at  426,  when  this  is  more  appropriately  transmitted  at  Squadron,  will  not  produce 
long-term  retention. 

It  is  important  to  note  that  for  a  conceptual  scaffolding  to  work,  it  must  be  supported  by  tools  that 
allow  strong  conceptual  understanding  to  be  stimulated  at  OTU  and  strong  knowledge-to-skill 
conversion  to  occur  during  OJT.  The  Smart  Boards  provide  just  such  a  ready  training  tool/aid 
for  the  development  of  stronger  conceptual  understanding  in  the  form  of  the  schematic  visual 
representation  of  aircraft  systems  presented  (usually)  in  the  upper  half  of  their  screen  displays. 
The  ability  of  the  student  to  manipulate  this  schematic  interactively,  and  to  receive  feedback 
about  the  results  of  their  control  actions,  promotes  a  strong  conceptual  understanding  of  how  and 
why  the  system  operates  as  it  does.  In  addition,  the  simulation  capabilities  of  the  Smart  Boards, 
in  combination  with  appropriate  cases  that  teach  operational  skill  in  the  relevant  contexts,  and  at 
the  pertinent  times,  allow  the  tool  to  achieve  the  aim  of  knowledge-to-skill  conversion  as  well. 

Implications.  The  division  of  training  labour  suggested  by  this  re-balanced  training  model 
implies  an  increased  reliance  on  skill  development  at  the  Squadrons,  which,  in  turn,  entails  a 
somewhat  higher  level  of  support  for  training  at  the  Squadron  level  than  currently  exists.  This 
support  should  come  in  the  form  of  transmission  of  the  Smart  Board  learning  tools  to  all 
squadrons,  accompanied  by  training  of  squadron  Standards  Officers  in  their  facilitator  role  within 
the  scheme  of  problem-based  (case-based),  self-directed  learning  (see  next  section).  Central  co¬ 
ordination,  monitoring,  and  evaluation  of  the  entire  training  model,  when  delivered  in  this  way, 
is  required,  and  should  be  based  most  appropriately  at  the  training  Squadron. 
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Problem-Based  Self-Directed  Learning 


One  influential  mechanism  for  fostering  a  stance  on  education  that  can  be  described  as  ‘life¬ 
long’  at  the  same  time  that  it  imparts  strong  concept  building,  is  the  use  of  problem  or  case- 
based,  self-directed  learning  [15,  17-19].  Because  there  are  many  ways  to  implement  what  is 
deemed  to  be  a  ‘problem-based’  and  ‘self-directed’  approach  in  education,  care  should  be  taken 
in  defining  these  terms.  When  this  is  done  it  will  be  discovered  that  not  all  elements  of  the 
approach  lend  themselves  to  easy  adoption  in  the  military  training  environment.  However, 
carefully  pruned,  the  techniques  of  this  approach  can  be  usefully  applied  both  to  the  OTU  and  to 
OJT  at  the  operational  squadron  levels.  This  approach  is  summarised,  and  an  attempt  is  made  to 
show  how  it  can  be  usefully  adapted  at  the  OTU  and  for  OJT. 

Problem-Based  Learning.  Classically,  problem-based  learning  as  been  described  as  involving 
the  presentation  to  students  of  specific  real  or  simulated  case  studies  in  a  particular  field  that 
present  a  problem  or  challenge  requiring  solution.  It  is  contended  that  via  such  ‘situated 
learning’  new  material  is  better  understood  and  retained,  because  it  is  learned  in  concert  with 
environmental  cues  to  be  encountered  in  the  real  situations  in  which  the  learning  will  be  relevant 

[20]. 

The  cases  are  meant  to  provide  a  springboard  for  learning,  and,  importantly,  for  learning  about 
learning.  Typically  directive  forms  of  instruction  such  as  the  chalk-and-talk  lecture  are  not  used 
with  this  approach  and  the  instructor’s  role  is  re-defined  as  being  one  of  an  intellectual  shepherd, 
mentor  or  facilitator.  Students  work  both  individually  and  in  small  groups,  to  define  actively  the 
issues  and  questions  to  be  addressed,  the  tasks  to  be  done,  and  the  information  to  be  discovered 
or  recruited.  In  this  way  the  method  is  self-directed  in  addition  to  being  problem-based.  After 
discovering  on  an  individual  basis  the  information  to  be  used,  trainees  then  work  together  with 
the  information  they  have  accumulated  to  synthesise,  discuss,  and  integrate  it,  and  to  make  final 
recommendations  about  the  problem  solution. 

Because  problems  are  multi-determined  and  multi-faceted,  the  information  sought  usually  must 
be  extracted  from  a  variety  of  disciplines  and  sources,  and  in  a  variety  of  ways.  It  is  contended 
that  this  is  a  learning  challenge  and  intellectual  environment  that  is  more  representative  of  the 
real-world  problems  and  performances  expected  of  employees  in  almost  every  field  once  they 
graduate  from  training.  And  it  is  also  asserted  that  such  ‘contextual  learning’  (i.e.,  learning  as 
needed,  just-in-time,  and  in  the  context  where  it  will  later  be  applied)  makes  the  learning  more 
enjoyable  and  more  effective  as  well. 

The  group-based  learning  portion  is  integral  to  the  process,  as  it  is  a  central  ‘method  of  work  ,  a 
method  again  often  representative  of  the  real  world  group  working  environments  in  which 
trainees  will  spend  most  of  their  professional  lives  [21,  22].  Thus,  the  knowledge  gained  about 
group  dynamics  in  the  group-centred  part  of  this  inquiry/leaming  process  also  transfers  to  the 
real  work  environment,  where  people  are  expected  more  often  to  perform  in  groups  than  alone 
(especially  in  highly  technical  work  environments). 

Co-operation  and  teamwork  skills  are  fostered  from  initial  training  onward  when  group-based 
learning  is  integrated  into  the  method  of  instruction,  as  people  become  aware  of  such  things  as: 


how  to  manage  differential  ability; 

how  to  contribute  both  to  teamwork  and  taskwork; 

how  to  participate  in  others’  learning; 

how  to  read  one’s  own  and  others’  emotional  reactions; 

how  to  build  trust; 


how  to  manage  and  resolve  conflict  constructively; 

how  to  give  feedback  to  one  another  usefully  and  sensitively; 

how  to  manage  difficult  people;  and 

how  to  instil  as  the  key  motive  of  team  life  the  performance  needs  of  getting  the  job  done 
safely  and  effectively,  rather  than  letting  fear  and  competition  drive  human  relations  and 
team  performance  in  either  the  training  or  real  world  environment. 


Self-directed,  problem-based  learning  is  usually  a  new  mode  of  learning  for  new  inductees  and  as 
such,  care  is  taken  at  the  outset  to  provide  training  to  trainees  in  the  method  of  training  itself  so 
hat  students  understand  the  approach  and  their  responsibilities  within  it.  During  this  initial 
training  trainees  are  engaged  in  learning  how  to  learn. 


miners  also  often  need  to  be  trained  in  the  mode  of  facilitation  required,  which  is  usually  quite 

ac.custofed  roles  as  ‘experts  professing’.  They  must  learn  in  explicit  terms 
how  to  help  students  learn  how  to  learn.  Trainers  are  asked  to  set  cases,  encourage  and  guide 
trainees  without  forcing  or  directing,  provide  a  role  model  of  critical  thinking  and  critical  self- 
examination,  and  assist  group  members  in  self-evaluation. 


Trainers  are  also  responsible  for  the  evaluation  of  each  member  of  the  group  and  for  shepherding 
group  processes  where  necessary.  Evaluations  can  be  formal  and  on-going  —  conducted  at  the 
end  of  each  session  —  or  can  be  performed  more  infrequently  at  mid-  and  end-of-course  And  of 
course,  objective  performance  standards  still  serve  as  essential  benchmarks  of  knowledge 
proficiency  against  which  hard  performance  evaluations  can  be  measured. 

Everything  in  this  approach  is  directed  toward  making  the  student  responsible  for  their  own 
learning  and  to  set  patterns  and  habits  of  learning  suitable  to  their  needs.  This  method  of 
operation  should  be  employed  continuously  throughout  their  professional  career.  However  it  is 
important  to  note  that  while  self-directed,  learning  is  neither  self-willed  nor  self-indulgent  in  this 
regime  Clear  expectations  for  training,  training  standards  and  performance  objectives  are  set 
and  met.  While  the  system  of  learning  delivery  is  intended  to  be  flexibly  adapted  to  individuals’ 
learning  styles  what  is  not  open  for  negotiation  is  the  requirement  to  produce  training  outcomes 
of  effective  and  efficient  subject  matter  knowledge  and  proficiency.  Therefore,  the  approach  is 
self-directed  and  student-centred  from  the  point  of  view  of  the  training  delivery  mechanism,  but 
the  programme  itself  is  not  self-directed.  Considerable  guidance  and  modelling  of  best-learning 
practices  is  one  of  the  key  elements  that  trainers  must  provide  in  their  re-defmed  role  and  the 
course  content  is  theirs  to  set  as  they  see  fit.  This,  in  summary,  is  the  ‘how’  of  problem-based 
learning  as  it  is  currently  conceived  and  practised. 


Overview  of  Domains  of  Knowledge  in  Problem-Based  Learning.  The  what’  or  content 
resulting  from  this  approach  is  usually  divided  into  three  intended  domains  of  impact’. 
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1.  A  knowledge  hase  is  grown:  Knowledge  consists  of  both  facts  and  concepts.  The  problem- 
based  learning  approach  tends  to  emphasise  the  development  of  sound  conceptual 
understanding  and  models  within  which  to  organise  facts  (i.e.,  conceptual  scaffolding),  as 
well  as  the  knowledge  of  how  to  gain  access  to  specific  information  as  required.  This  places 
it  in  keeping  with  the  training  model  proposed  for  the  OTU,  wherein  a  greater  emphasis  on 
conceptual  understanding  is  recommended  than  is  currently  practised. 

2  The  second  domain  of  learning  is  skill:  In  this  case  this  will  include  not  only  systems 
operational  skill,  procedural  skill,  and  psychomotor  skill,  but  also: 

•  critical  appraisal  skill,  such  as  system  situation  awareness,  the  application  of  judgement,  a 
questioning  attitude,  the  ability  to  solicit  and  weigh  evidence  (like  the  quality  of  current 
flight  data),  system  diagnostic  and  trouble-shooting  skill,  etc.;  and 

•  learning  skills,  such  as  the  ability  to  self-identify  deficiencies  in  one’s  own  systems 
knowledge,  the  ability  to  correct  deficiencies,  knowledge  of  resources  available  to  correct 
knowledge  deficits,  and  understanding  the  importance  of  constantly  renewing  one’s  base 
of  knowledge  (i.e.,  motivated  remediation). 

3  Finally,  problem-based  self-directed  learning  behaviours:  This  is  a  third  kind  of  domain 

knowledge _ namely  the  acquisition,  development  and  maintenance  of  personal  qualities 

and  attitudes  appropriate  to  the  profession’s  roles  and  tasks.  This  knowledge  includes: 

•  team  leadership  and  followership  skills; 

•  mentoring  skills  modelled  from  the  examples  provided  by  the  intellectual  shepherds  — 
the  trainers;  and 

•  a  motivated  attitude  of  life-long  learning  —  continual  self-improvement. 

Thus,  in  total,  the  process  is  intrinsically  oriented  toward  the  kind  of  personal  and  teamwork 
development  required  of  left-seaters  in  their  professional  conduct  inside  the  cockpit  and  outside 

of  it  as  well. 

The  Case  for  Case-Based  Training.  The  self-directed  learning  philosophy  need  not  be 
problem-based.  It  is  important  to  review  from  a  cognitive  and  methodological  perspective,  why 
case-based  (nee:  problem-based)  teaching  methods  are  likely  to  be  effective  in  meeting  the  goal 
of  fostering  expertise  in  an  accelerated  time  frame  within  the  CC-130  training  program.  Here  is 
why  knowledge  is  better  transformed  into  operational  skill  using  cases. 

Formal  conversion  training  is  often  designed  such  that  operators  receive  a  level  of  training 
necessary  to  produce  ‘competent’  performance,  that  is,  a  performance  level  sufficient  to  produce 
safe  system  operation  [23].  At  the  end  of  formal  conversion  training,  pilots  can  probably  be  best 
described  as  ‘trained  novices’  [24]  rather  than  as  anything  approaching  experts.  At  this  point 
their  acquired  knowledge  is  unlikely  to  be  well-differentiated  or  well-integrated,  characteristics 
classically  associated  with  expertise.  On  a  continuum  that  ranges  from  compartmentalised 
knowledge  to  integrated  knowledge,  the  structure  of  the  systems  mental  models  held  by  trainees 
at  this  point  in  their  training  is  likely  to  be  classed  towards  the  fractionated,  or 
compartmentalised,  end  of  the  continuum.  The  training  challenge  presented  by  systems 


knowledge  acquisition  is,  of  course,  to  devise  tools  and  methods  of  training  delivery  that  will 

allow  trainees  to  progress  along  this  continuum  of  integration  toward  the  more  expert  end _ in 

the  absence  of  real  world  experiences  with  all  likely  or  unlikely  systems  situations. 

With  reference  to  specific  kinds  of  knowledge  to  be  trained,  pilots  will  have  three  sorts  of 
understandings  represented  in  their  developing  mental  models  of  aircraft  systems  at  this  point  in 
their  post-conversion  course: 

•  declarative  knowledge, 

•  procedural  knowledge,  and 

•  operational  skill  [25], 


Declarative  knowledge  is  the  factual  knowledge  possessed  about  the  system,  things  like  its 
components,  controls,  and  functions.  Procedural  knowledge  refers  to  the  rule-based  knowledge 
possessed,  such  as  how  to  operate  the  system  and  how  to  perform  sub-component  tasks. 
Operational  skill  is  the  ability  to  integrate  the  first  two  forms  of  knowledge,  the  declarative  and 
the  procedural,  in  a  goal-directed,  multi-task,  time-  and  resource-constrained  environment,  in 
other  words,  in  the  kind  of  dynamic  decision  making  environment  that  the  CC-130  flightdeck 
represents.  It  also  includes  executive  function  sorts  of  knowledge,  such  as  recognising  when 
activities  are  required  (situational  awareness,  implicational  thinking,  prioritisation,  etc.)  and 
knowing  how  to  co-ordinate  these  activities  to  meet  the  demands  of  real-time  operations  (e.g., 
pre-planning,  managing  time,  allocating  aircraft  and  team/human  resources,  etc.). 

Expertise  is  most  related  to  the  last  of  these  three  knowledge  contents  —  the  operational 
application  of  knowledge  in  real  time.  Studies  have  demonstrated  that  operators  matched  on 
their  declarative  and  procedural  knowledge  (as  assessed  by  test  scores),  but  differing  in  expertise, 
show  marked  differences  in  their  operational  performance  effectiveness  [26].  Trained  novices 
typically  exit  from  formal  training  with  incomplete  declarative  knowledge  and  incomplete  and 
only  loosely  integrated  procedural  knowledge,  yielding  a  ‘buggy’  cognitive  representation  of 
each  system  [27]. 


Usually,  expertise  is  obtained  over  time  —  through  the  accumulation  of  real  world  experiences 
that  are  integrated  into  the  basic  mental  conceptual  model.  By  experiencing  a  wide  range  of 
situations,  acting  to  control  systems  within  those  situations,  and  receiving  feedback  about  the 
results  of  control  actions  attempted,  pilots’  declarative  and  procedural  knowledge  becomes 
integrated  into  operational  skill.  Unfortunately,  OJT  sometimes  does  not  expose  all  pilots  to  all 
pertinent  situations,  so  that  line  flying  accumulated  over  a  period  of  time  does  not  always  build 
expertise  sufficiently.  In  addition,  after-action  or  after-event  reviews  may  be  missed  [28], 
insufficiently  detailed,  or  informal  at  the  Squadron  level,  causing  valuable  training  lessons  to  be 
foregone  and  learning  opportunities  to  be  lost. 

Herein  lies  the  great  potential  of  a  case-based  self-directed  training  system  for  use  primarily  at 
OJT.  By  collecting  and  articulating  in  the  form  of  case-based  learning,  a  widely  representative 
sample  of  typical  actions  and  Standard  Operating  Procedures  (SOPs),  as  well  as  known  incidents, 
accidents,  anomalies,  and  typical  faults,  one  can  with  simulation  and  a  repertoire  of  cases, 
approximate  the  breadth  and  depth  of  training  that  would  normally  occur  over  a  more  extended 
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period  of  time.  Encapsulating  the  organisation’s  knowledge  in  this  way  also  serves  to  preserve 
its  organisational  memory  of  ‘war  stories’  and  ‘hangar  flying’  — its  informally  archived  cases. 

Note  that  training  success  will  depend  in  large  measure  upon  the  selection  of  appropriate  training 
scenarios  under  a  case-based  approach.  ‘Appropriate’  entails  a  choice  of  case  scenarios  that 
provides  the  proper  environmental  contexts  and  cues  that  reflect  the  way  systems  knowledge  is 
actually  used  in  that  system’s  domain.  If  chosen  or  built  wisely,  the  simulator  cases  will  promote 
the  integration  of  knowledge  and  enhance  operational  skill  [29].  By  trading  cases  with  other 
fleets  operating  the  C-130  airframe,  an  even  fuller  repertoire  of  situations  can  be  represented. 

Problem-Based  Learning’s  Best  (and  Appropriate)  Practices  for  OTU/OJT.  It  is  recognised 
that  426  Squadron  OTU  currently  employs  a  variant  of  a  problem-based  learning  component  in 
its  use  of  case  studies.  A  radical  re-design  of  the  training  system,  in  favour  of  a  completely 
problem-based  approach,  is  not  suggested.  Chalk-and-talk  lectures  and  other  forms  of  teaching 
should  continue  to  have  their  proper  place  in  the  methods  of  training  or  delivery  used. 

However,  for  the  future,  when  such  training  challenges  as  the  Avionics  Update  Programme  arise, 
it  is  believed  that  the  benefits  of  life-long  learning  and  self-study  via  self-directed  learning  will 
become  more  necessary  as  mechanisms  to  help  aircrew  remain  current  in  an  increasingly  systems 
knowledge-intense  environment.  While  the  role  of  the  training  instructor  remains  important  and 
will  not  be  diminished  in  this  new  scheme  (in  fact,  it  is  enhanced  and  more  appropriately 
deployed),  students  must  be  given  better  resources,  skills,  and  the  motivation  for  self-directed 
learning  and  self-paced  study  as  an  additional  avenue  toward  continuous  performance 
improvement.  Further,  they  must  be  exposed  to  the  kinds  of  group-based  learning  which  mimic 
their  work  environments  on  the  flight  deck.  Once  the  group-based  format  is  acquired  during 
ground  school  at  426,  it  will  be  easily  recapitulated  as  a  ‘method  of  work’  in  the  operational 
environments  at  the  squadrons.  Moreover,  it  has  great  potential  as  a  distance-based  group 
method  using  suitable  real  time  communication  across  computer  networks,  allowing  teams  to  be 
composed  from  members  at  a  distance  —  such  as  from  different  squadrons. 

Certain,  limited  aspects  of  the  problem-based  learning  approach  described  above,  can  be 
translated  into  training  practices  that  will  have  a  positive  payoff  for  the  Hercules  pilot 
community.  In  fact,  existing  elements  of  the  OTU’s  current  lessons,  structures  and  philosophy 
lend  them  naturally  to  a  problem-based  approach.  Class  sizes,  for  example,  are  already  small, 
much  in  keeping  with  the  5-6  member  groups  recommended  by  most  problem-based  programs. 
Lessons  are  already  designed  that  focus  on  systems  failures,  a  good  springboard  for  the  problem- 
based  mode.  And  a  compendium  of  ‘war  stories’  (although  as  yet  undocumented)  can  also  act  as 
excellent  grist  for  the  problem-solving  mill. 

The  ‘limited  aspects’  of  the  approach,  referred  to  above  as  appropriate  areas  for  adoption,  are 
summarised  in  the  following  list: 

•  it  is  not  recommended  that  the  entire  training  effort  be  converted  to  this  format  —  lectures 
are  still  the  most  efficient  mode  of  rapid  information  transfer  (except  for  CBT)  when  this  is 
what  is  required,  for  example  in  early  learning; 
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•  more  study  must  be  given  as  to  which  specific  lessons  are  best  delivered  in  the  problem- 
based  mode; 

•  implementation  should  be  phased,  and  slow  to  moderate  in  speed; 

•  proper  training  in  the  method  for  trainees  must  be  supplied  — such  training  is  essential  in 
teaching  requisite  skills  for  problem-based  learning  and  in  setting  expectations  for  training 
and  self-efficacy  —  the  latter  factors  have  been  shown  to  be  instrumental  in  the  effectiveness 
of  training  [16]; 

•  proper  training  for  trainers  must  be  executed,  enabling  trainers  to  assume  their  new  roles  as 
facilitators,  mentors.  Subject  Matter  Expert  (SME)  models,  resources  for  self-evaluation  etc.- 
and 

•  objective  evaluations  must  be  developed  (subjective  forms  of  evaluation,  inappropriate  to  the 
military  training  context,  are  often  used  in  applications  of  problem-based  learning). 

At  present,  the  times,  places,  and  choice  of  learning  path  used  by  the  student  are  still  largely  in 
the  hands  of  the  training  system.  The  results  of  problem-based  learning  approaches  indicate  that, 
within  limits  and  with  considerable  guidance  at  times,  trainees’  learning  is  more  effective  when 
they  have  some  control  over  these  factors.  Students  learns  best  when  they  can  identify  their 
learning  needs,  select  learning  methods  that  suit  them  best,  select  the  location  (time  and  place) 
where  learning  is  maximised,  and  learn  to  identify  through  appropriate  unsolicited,  and  later 
requested,  feedback,  whether  or  not  they  are  achieving  their  goals. 

A  suitable  tool  for  self-paced  study  is  needed  that  would  allow  such  flexibility  in  the  learning  of 
systems  knowledge  and  in  the  continual  upgrading  required  to  remain  current  on  the  complex 
systems  contained  in  the  CC-130  at  present  and  in  the  future.  The  basics  of  such  a  self-study  tool 
already  exists  in  the  form  of  the  Smart  Board  system  simulation  interactive  CD  ROMs.  While 
currently  this  tool  is  used  primarily  in  the  classroom  at  the  OTU,  it  can  be  amended  for  use  as  a 
stand-alone  self-directed  learning  resource  made  available  at  the  Squadrons  and  for  use  at  home. 
As  its  hardware  requirements  are  currently  within  the  abilities  of  most  home  PC  systems  on  the 
market  today  (and  in  the  future),  and  as  the  software  upgrades  to  these  packages  needed  to 
facilitate  self-paced  study  do  not  require  additional  PC  system  capabilities,  this  tool  has  excellent 
self-directed  learning  potential. 

To  fully  realise  the  learning  potential  of  the  Smart  Board  ‘part  task  simulators’,  several  ways  in 
which  this  technology  can  be  leveraged  are  suggested  (see  next  sections),  for  example,  by 
combining  them  with  video  re-creations  of  documented  incidents/accidents  and  known  problems, 
by  adding  an  Awareness-Implications-Planning  (AIP)  overlay  module  to  begin  to  explicitly 
teach  AIP  principles  as  habits  of  thought,  and  by  adding  some  intelligent  tutoring  capabilities 
(see  below). 

Implications.  To  summarise,  the  addition  of  a  problem-based  learning  focus  in  the  OTU  and 
beyond  carries  with  it  several  implications: 

•  the  requirement  to  train  trainees  in  this  method  of  work  at  OTU  (initially  at  the  conversion 

course  or  earlier  for  pipeliners),  so  that  they  may  continue  this  form  of  education  at  OJT _ 

this  is  critical  to  its  success  in  the  creation  and  effective  management  of  work  teams; 
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•  the  requirement  to  train  the  trainers  in  this  method  both  at  the  OTU  and  at  the  operational 
units; 

•  the  need  to  create  self-paced  study  tools  (via  software  extensions  of  Smart  Board),  and 

•  the  need  to  create  and  manage  a  compendium  of  training  cases. 


Effective  Use  of  Interactive  Multi-Media  for  OTU/OJT  and  Self-Study 

Recent  trends  in  technical  education  point  squarely  to  multi-media  packages  as  a  currently 
preferred  major  mechanism  of  course  delivery.  Efficacy  studies  show  the  reduced  knowledge 
acquisition  times  and  enhanced  knowledge  retention  levels  achievable  via  these  packages,  be 
they  traditional  computer  based  training  (CBT)  or  low  to  mid-fidelity  simulations  [30,  31]. 
Another  chief  advantage  of  multi-media  is  its  potential  for  self-study  application  outside  the 

classroom. 

Specific  ways  in  which  the  Smart  Boards  may  be  leveraged  to  improve  them  even  further  are 
now  reviewed.  Following  this,  a  recent  development  in  intelligent  tutoring  is  described  —  one 
that  employs  cases  as  method  within  a  multi-media  environment,  in  keeping  with  our  emphasis 
on  problem-  or  case-based  teaching  methods. 

Improvements  to  the  Smart  Boards.  Smart  Board  modules  are  excellent  training  aids,  and 
early  anecdotal  evidence  of  their  efficacy  regarding  learning  speed  and  retention  from  426 
Squadron  is  extremely  promising  (personal  correspondence).  It  is  understood  that  their  initial 
development  was  for  the  training  of  technicians,  but  it  is  believed  that  they  can  be  even  further 
enhanced  to  achieve  the  kind  of  pilot  knowledge  and  skill  acquisition  objectives  enumerated  in 
this  Volume.  They  should  therefore  be  finished  on  a  fast  track  for  all  systems  not  currently 
represented  in  this  format,  and  appropriate  applications  should  be  developed  for  the  conversion 
course,  pilot  upgrade  to  left  seat  course,  and  Flight  Engineer’s  (FEs’)  course  at  the  OTU,  as  well 
as  for  applications  at  the  Squadron  level.  In  the  terms  already  discussed  for  effective,  efficient, 
and  evaluatable  training,  the  Smart  Boards  lend  themselves  readily  for  application  because: 


•  the  schematics  provide  a  visual  conceptual  scaffolding  of  the  kind  whose  importance  has 
been  emphasised  as  the  central  learning  objective  for  pilots  at  426  Squadron; 

*  •  the  Smart  Board  software  is  easy  to  distribute  to  Squadrons,  hardware  infrastructure 
notwithstanding  —  the  full  system  dynamic  representation  in  the  ‘lower’  half  of  the  display 
is  an  effective  detail/skill  developer  that  can  be  resourced  repeatedly  at  the  Squadron  level  to 
enhance  the  conceptual  underpinning  imparted  at  ground  school  and  to  integrate  declarative 
and  procedural  knowledge  into  operational  skill; 

•  with  relatively  simple  enhancements,  the  AIP  formulation  can  be  integrated  with  current 
Smart  Board  representations  (see  next  section)  —  the  focus  again  will  be  on  system 
implications  and  system  interdependencies;  and 

•  Smart  Board  feedback  functionality  can  be  incorporated  anywhere  along  a  continuum  from 
non-intelligent  interfaces  (e.g.,  pop-up  windows  with  verbal  answers)  to  intelligent  ones  (e.g-, 
intelligent  tutors)  —  because  an  eventual  goal  is  to  allow  self-directed  learning,  it  is 
recommended  that  at  least  some  measure  of  intelligence  is  put  behind  the  tutoring  system. 


The  potential  for  self-study  using  the  Smart  Board  is  obvious  (see  last  point).  Some  direction 
will  have  to  be  given  as  to  the  appropriate  (or  standards-based)  level  at  which  the  material  is 
expected  to  be  learned  for  a  given  crew  position,  but  the  potential  for  learning  to  whatever  depth 
is  desired  by  pilots  should  be  preserved.  The  following  improvements  are  recommended. 

*  Allov^  aircrew  to  interact  with  systems  simulations:  when  illustrating  cases,  having  a 
dynamic  representation  of  aircraft  systems  in  real  time  in  the  form  of  a  Smart  Board  display 
as  well  as  a  real-time  video  simulation  of  cockpit  events  (played  on-screen  eventually  as 
hardware  is  upgraded,  and  via  a  parallel  presentation  medium  initially),  will  be  a  powerful 
aid  to  training.  The  instructor  should  be  able  either  to  stipulate  the  end  point  of  the  dynamic 
system  or  to  render  ineffective  certain  student  actions,  so  as  to  be  able  to  simulate  real  aircraft 
cases  of  incidents  and  accidents  where  pilot  control  actions  proved  ineffective. 

*  -Allow  monitoring,  evaluation  and  feedback  of  student  actions:  whether  in-course  or  in  self- 
study  mode,  the  smart  boards  can  be  engineered  to  come  equipped  with  ‘canned’  starting 
system  parameters.  By  recording  obvious  indicators  such  as  pilot  control  actions  and  their 
sequences,  reaction  times,  and  responses  to  open-ended  queries  (e.g.,  problem  awareness  and 

lagnosis,  implications,  pre-planning,  knowledge  of  system  interactions,  etc.),  instructors  (or 
an  intelligent  tutor)  can  evaluate  a  student’s  systems  knowledge.  Such  measures  taken  over 
time  would  allow  tracking  of  student  progress.  Used  as  a  formal  evaluation  method 
normative  data  could  be  collected  about  fleet-wide  trends  using  a  non-jeopardy  data  base 
Especially  in  self-study  mode,  feedback  to  the  student  can  be  given  in  a  way  that  provides 
raw  data  for  the  development  of  meta-knowledge  (knowledge  about  the  state  of  one’s 
knowledge),  an  important  precursor  of  expertise  and  objective  self-evaluation  about 
proficiency  in  knowledge  and  habits  of  thought. 

Intelligent  Tutors  for  Simulation.  It  is  encouraging  to  have  recently  found  a  training  tool  with 
many  of  the  formal  properties  that  have  been  reviewed  above  as  being  desirable  best  training 
practices.  This  tool  was  conceptualised  independently  of  this  review,  but  is  convergent  in  its 
recommended  methods  for  training  delivery.  It  is  reviewed  now  in  some  detail,  noting  at  the 
outset  a  number  of  ways  in  which  it  is  similar  and  number  of  ways  in  which  it  differs  from  the 
current  proposals  for  renewed  training  delivery. 


Chappell  and  Mitchell  [27]  recently  described  an  architecture  for  a  computer-based  training 
system  for  operators  of  complex  dynamic  systems,  the  Case-Based  Intelligent  Tutoring  System 
(CB-ITS).  This  can  serve  as  a  conceptual  roadmap  for  leveraging  the  investments  made  to  date 
by  8  Wing  in  the  Smart  Board  systems  knowledge  delivery  modules.  It  is  similar  to  the  approach 
recommended  in  this  Volume,  as  follows: 

•  it  is  intended  to  help  pilots  enhance  and  maintain  expertise; 

•  it  is  inherently  case-  or  problem-based,  providing  users  with  normal  as  well  as  unusual 
operating  situations; 

•  it  emphasises  the  acquisition  of  sound  conceptual  underpinnings  for  systems  knowledge _ 

the  ‘how’  and  ‘why’  of  systems  functioning,  not  just  the  ‘what’; 

•  it  includes  a  simulator  that  allows  students  to  practice  translating  their  procedural  knowledge 
into  operational  skill; 


•  it  includes  an  intelligent  tutor  with  variable  ability  to  provide  intelligence  and  instructional 
control  as  a  function  of  student  needs;  and 

•  it  is  inherently  closed-loop  and  systems-based  (e.g.,  in  its  use  of  feedback  to  the  student  to 
help  null  the  error  signals  while  learning). 

Chappell  and  Mitchell’s  approach  is  different  to  the  approach  to  instructional  delivery 
recommended  in  this  Volume,  in  the  following  ways: 

•  it  is  designed  primarily  as  a  self-directed  form  of  OJT,  whereas  the  approach  recommended 
for  the  CC-130  would  marry  it  to  both  OTU  and  OJT  in  terms  of  a  common  approach  and 
method  of  work; 

•  it  does  not  integrate  an  AIP  framework  (see  next  section); 

•  it  does  not  integrate  teamwork  into  the  method  of  work; 

•  it  has  less  emphasis  on  developing  high  level  self  monitoring  skills  and  self-evaluation;  and 

•  it  uses  a  different  simulation  platform. 

These  points  stated,  the  utility  of  intelligent  tutoring  systems  (ITSs)  lies  in  their  ability  to  tailor 
instruction  to  the  needs  of  individual  students.  To  do  so,  such  systems  contain  four  components: 

•  a  domain  expert, 

•  a  student  model, 

•  the  instructional  method,  and 

•  the  tutor  interface  [32,  33]. 

The  domain  expert  is  a  model  representing  knowledge  that  characterises  expert  performance, 
including  declarative  and  procedural  knowledge,  and  operational  skill.  The  student  model 
represents  the  tutor’s  changing  knowledge  of  the  student’s  knowledge.  By  comparing  at  any 
point  the  domain  model  with  the  student  model,  the  tutor  selects  teaching  strategies  such  as 
cases,  illustrations,  hints,  coaching,  etc.,  that  will  close  this  expert-student  gap.  Such  contents 
are  delivered  via  interaction  with  the  student  using  the  tutor  interface. 

An  ITS  is  intrinsically  closed  loop  or  cybernetic  in  nature,  in  keeping  with  the  overall  theme  we 
have  struck  regarding  systems  theory.  In  attempting  to  null  the  ‘error  signal’  between  student 
and  expert,  the  ITS  exerts  control  actions  on  the  student.  As  a  student  becomes  more  advanced, 
the  ITS  exerts  less  control  and  models  its  expertise  to  a  decreasing  degree  along  a  continuum 
ranging  from  active  prompting,  to  providing  feedback  when  it  notes  an  error  or  anomaly,  to 
minimal  intervention  (i.e.,  only  providing  updates  on  systems  at  checkpoints,  when  errors  are  no 
longer  recoverable). 

Collins  [20]  suggests  that  an  ITS  delivers  a  form  of  ‘cognitive  apprenticeship’  to  the  student  by 
demonstrating  expert  performance  in  the  context  of  realistic  task  scenarios.  In  doing  so  it  helps 
students  to  better  integrate  knowledge  of  systems  and  procedure  into  operational  skill  because 
the  knowledge  of  systems  required  for  safe  and  efficient  flight  performance  that  is  illustrated 
occurs  in  a  multi-task,  real-time  simulation.  Cognitive  apprenticeship  has  been  used  with  some 
success  in  several  training  systems  that  teach  performance  and  decision  making  in  naturalistic 
decision  making  environments  with  complex  systems  [25,  34]. 
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During  simulation,  CB-ITS  allows  students  to  make  the  implicit  knowledge  contained  in  its 
expert  model  explicit  at  any  point  in  time.  The  student  simply  must  ask  why  the  steps  in  an 
activity  may  have  been  prescribed  by  the  tutor  to  receive  reasons  that  describe  the  pertinent 
declarative  and  procedural  knowledge  and  integrate  it  into  the  situation  in  progress.  This  too 
promotes  the  development  of  operational  skill  and  perhaps  meta-awareness  of  the  state  of  one’s 
knowledge  as  well. 

Other  expert  systems  offer  similar  capabilities,  and  may  even  allow  better  development  of 
students  expertise  and  meta-knowledge.  For  example,  the  expert  system  knowledge  elicitation 
package  EXSYS  prompts  students  with  questions  about  the  state  of  their  knowledge  [35]. 
During  this  process  students  may  at  any  point  ask  why  the  question  itself  is  being  asked  by  the 
system.  A  tutor  that  describes  why  it  is  making  queries,  in  the  manner  and  order  in  which  it  is, 
can  usefully  model  the  expert  s  thinking  process.  The  ability  to  reason  about  novel  situations  is  a 
hallmark  of  expertise,  and  among  the  most  important  things  to  model  and  learn  along  the  road  to 
expert  knowledge  is  the  sequence  of  appropriate  questions  to  ask.  The  implicit  causal  model  of 
systems  operations  held  by  the  expert  that  allows  reasoning,  a  model  that  one  would  like  to  have 
students  learn,  is  often  well  exposed  by  observing  the  expert’s  process  of  inquiry,  that  is,  the 
processes  for  generating  new  knowledge. 

In  summary,  several  expert  knowledge  representation  software  simulation  packages  exist  that  can 
be  usefully  built  into  the  Smart  Board  system.  When  incorporated  as  intelligent  tutors,  these 
have  the  potential  to  provide  effective  and  efficient  training  via  self-study. 

Implications.  Implied  in  the  use  of  multi-media  packages  is  that  8  Wing  should  formally 
evaluate  their  efficacy  (as  has  been  begun  for  the  Smart  Board  software).  Chief  downsides  of 
multi-media  include,  of  course,  the  initial  development  costs  where  customisation  is  needed. 
Corporations  are  placing  such  materials  on  their  intranets  for  distance  education  purposes  and 
self-study,  implying  additional  costs  for  IT  positions  to  support  it,  authoring  software  and 
instructor  training,  course  management  software,  etc.  The  integration  of  an  intelligent  tutor  will 
require  an  initial  phase  of  expertise  extraction  and  system  representation  in  the  domain  expert 
model  built  for  each  aircraft  system. 


Development  of  Superior  Mental  Models  of  Aircraft  Systems 

Pilots  in  the  CBAG  validation  study  fell  down  on  systems  knowledge  in  two  ways:  first,  because 
they  had  trouble  in  the  initial  recognition  or  diagnosis  of  problems  and  solutions,  or  second, 
because  they  failed  to  adequately  take  into  account  and  plan  ahead  for  possible  implications  of 
the  systems  problems  they  encountered.  By  teaching  as  a  mental  heuristic  the  Awareness- 
Implications-Planning  approach  —  namely,  how  to  be  more  aware  of  developing  system  states, 
how  to  create  a  habit  of  thought  to  consider  the  future  implications  of  the  systems  problems 
experienced,  and  how  to  pre-plan  for  various  systems  and  inter-systems  contingencies  —  pilots 
will  come  to  routinise  in  their  thinking  the  very  kind  of  diagnostic  abilities  they  most  need  to 
solve  systems  problems. 


23 


The  longer  term  view  of  this  approach  is  to  state  that  it  is  consistent  with  the  emphasis  placed 
elsewhere  (this  report,  see  Volume  5)  on  the  theme  of  cockpit  decision  making,  especially  on  the 
part  of  ACs.  In  this  case,  AIP  contributes  to  decision  making  about  systems  in  particular,  and 
about  the  entire  mission,  as  systems  problems  are  considered  in  light  of  the  big  picture. 

Implications.  One  way  to  instil  an  AIP  mental  approach  to  systems  considerations  is  to  query 
pilots  in  mid-simulation  about  what  they  are  thinking  about  the  systems  they  are  operating.  This 
would  require  the  capability  for  the  instructor  to  stop  the  simulation  to  make  the  inquiry  (or 
allow  the  pilot  to  record  their  thinking  in  response  to  a  pop-up  window  query).  As  this  might 
prove  disruptive,  an  alternate  tack  is  to  make  the  AIP  inquiry  part  of  all  after-action  and  post¬ 
flight  reviews  (including  part-task  and  simulator  training),  and  consider  making  it  overtly 
manifest  via  inclusion  in  training  checklists,  say  during  the  take-off  and  approach  briefs.  At 
minimum,  a  simple  extension  of  instructors’  current  case  scenarios  that  would  ask  pilots  about 
systems  implications,  both  in  the  future  and  for  other  systems  as  new  faults  are  encountered, 
would  go  far  in  making  this  higher  order  mental  model  checklist  habitual. 

Appropriate  and  valid  objective  assessment  instruments  and  standards  need  to  be  developed  that 
would  make  such  implicational  thinking  and  pre-planning  activities  subject  to  formal  evaluation. 
Feedback  will  be  needed  to  allow  students  to  develop  an  awareness  of  their  strengths  and 
weaknesses  in  these  higher  order  cognitive  skills  (especially  in  self-study  mode),  and  the  fleet 
can  be  monitored  squadron-wide  to  index  the  development  of  such  safety  skills  in  the  non¬ 
jeopardy  database. 


The  Need  for  Evaluation  at  Three  Levels 

Consistent  with  the  general  approach  and  framework  of  control  systems  theory,  students  also 
need  appropriate  feedback  in  order  to  evaluate  their  progress  and  achieve  learning  (i.e.,  null  the 
error  signals  they  are  receiving  about  whether  their  actions  on  the  environment  are  perceived  to 
move  them  toward  their  goals).  Mechanisms  for  this  are  many.  The  need  for  evaluation  to  occur 
on  at  least  three  distinct  levels  has  been  identified,  namely: 

1.  Trainee  Self-Evaluation 

2.  OTU/OJT  Formal  Student  Evaluation  and  Organisation-Wide  Monitoring  and  Evaluation; 
and 

3.  Formal  Efficacy  Study 

Trainee  Self  Evaluation.  The  development  of  meta-awareness  is  a  main  goal  of  self-evaluation. 
This  refers  to  the  developing  ability  of  a  student  to  think  about  their  own  learning  —  its 
completeness,  strengths  and  weaknesses,  habits  of  thought,  barriers,  etc.  Only  when  a  student  is 
capable  of  diagnosing  what  they  do  not  know,  and  is  motivated  to  correct  this  deficiency,  can 
they  be  said  to  be  a  self-directed  learner.  This  kind  of  meta-awareness  is  a  demonstrated 
correlate  (and  most  likely,  cause)  of  independent  thinking  and  expertise.  It  must  be  explicitly 
modelled  as  a  habit  of  thought  by  trainers  instructed  about  its  importance,  and  themselves  taught 
how  to  articulate  their  tacit  expertise. 
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OTU/OJT  Student  Evaluation  and  Organisation-Wide  Monitoring.  In  preparation  for 
documenting  the  efficacy  of  the  suggested  modifications  to  the  training  system  in  general  (e  g 
use  of  more  problem-based  teaching  with  cases)  and  the  Smart  Board  modules  in  particular  (use 
of  cases,  self-study,  intelligent  tutor,  etc.),  standards  of  training  will  need  development  and 
evaluations  of  student  performance  against  those  standards  will  have  to  be  performed  Student 
evaluation  in  quantitative  terms  must  be  part  of  the  revised  training  package,  along  with 
mechanisms  for  student  feedback  about  performance.  Tracking  pilot  performances  in  the  area  of 
systems  knowledge 

•  to  establish  a  baseline,  and  through  time,  at  exit  from  the  various  pilot  training  courses  at 
OTU,  as  well  as 

®  via  the  creation  of  a  no-jeopardy  data  base  build  on  spot-testing  at  OJT  or  appropriately 
systems  knowledge-intensive,  event-based,  full-motion  simulator  scenario  tests  delivered 
during  continuance  training, 

will  serve  as  mechanisms  for  individual  and  fleetwide  evaluation,  respectively  of  svstems 
knowledge  proficiency.  ’  3 

Formal  Efficacy  Study.  Finally,  using  the  assessment  instruments  and  standards  developed  for 
proficient  systems  knowledge  for  each  system,  a  formal  study  of  the  efficacy  of  this  training  as 
compared  to  either:  & 

1.  a  ‘pre-improvemenf  baseline,  or 

2.  another  CC-130  training  environment, 

should  be  conducted.  The  first  option  is  preferable  empirically,  and  because  the  main 
mechanisms  for  the  re-engineered  course  delivery  are  the  case-based  approach  and  the  Smart 
Board  technologies,  these  are  the  main  avenues  of  potential  improvement  and  therefore  should  be 
the  main  locus  of  evaluation.  If  their  efficacy  can  be  demonstrated,  this  will  be  powerful 
evidence  for  the  teaching  methods  recommended.  This  training  product  can  then  be  ‘sold’  to 

other  CC-130  fleets  world- wide  and  to  the  CF  fleet- wide. 

Implications.  Again,  if  implemented  at  8  Wing,  self-evaluation  training  would  require  feedback 
to  students  and  the  attendant  support  for  it,  especially  via  426  and  Squadron  training/standards 
experts  as  well  as  by  appropriate  software-based  feedback.  New  standards  will  need  to  be 
developed  for  evaluation  purposes  (e.g.,  for  AIP),  and  a  fleet  management  system  and  no- 
jeopardy  database  to  accumulate  fleet  wide  information  to  track  trends  in  systems  knowledge 
community  wide  should  be  instituted.  A  formal  efficacy  study  may  require  the  co-operation  of 
another  C- 130  fleet  for  comparative  purposes.  ~ 
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RECOMMENDATIONS 


In  light  of  the  best  educational  and  training  practices  as  enumerated  in  the  six  themes  in  the 
previous  Section,  and  building  on  the  considerable  strengths  of  the  current  training  system,  the 
following  recommendations  for  systems  knowledge  training  improvements  for  Phase  in  of  this 
project  are  made.  These  have  been  made  in  light  of  the  constraints  of  staffing  complements, 
fiscal  restraint,  organisational  restructuring,  and  existing  computer  hardware  and  software 
infrastructures,  and  reflect,  it  is  believed,  realistic  goals. 

L  Distribute  more  formal  training  to  the  Squadrons.  Strong  conceptual  foundations  in  aircraft 
systems  should  be  an  increased  focus  of  training  to  compensate  for  experience  loss,  and  are 
most  appropriately  conveyed  during  the  limited  exposures  pilots  get  to  426  Squadron  (rather 
than  an  over-emphasis  on  detail  in  that  venue).  Convey  more  of  the  detail  at  Squadron, 
where  knowledge  becomes  skill,  and  do  so  more  regularly  and  formally  via  scheduled 
training,  evaluation,  etc.  The  overall  plan  should  be  orchestrated  by  426,  in  consultation  with 
Squadrons  after  an  assessment  of  their  needs.  An  increased  role  for  OJT  may  imply  more 
training  of  Standards  Officers  in  educational  theory,  use  of  case-based  instruction,  computer 
literacy,  methods  of  evaluation,  etc.  However,  the  results  of  this  process  will  be  that 
Squadrons  will  have  more  of  a  voice  in  creating  the  kind  of  operations-specific  training  of 
systems  knowledge  they  require  on  the  line,  and  will  ‘own’  the  training  to  a  greater  degree. 


2-  Use  more  case-based  teaching,  supported  by  video  simulations  of  good  as  well  as  poor 
systems  handling.  Case-based  teaching  is  both  efficacious  and  enjoyable.  It  can  be  further 
enhanced  in  two  ways.  First,  it  may  be  paired  with  video  simulations  of  real  aircraft 
incidents/accidents,  as  well  as  illustrations  of  best  practices.  The  videos  would  be  created 
using  incident/accident/best  practices  transcripts  as  a  basis  and  using  role-playing  reservists 
or  retirees  as  actors.  This  would  require  use  of  the  full-motion  simulator  for  videotaping 
(using  typically  the  E  and  F  sessions  of  the  training  programme).  Second,  cases  can  be 
embedded  within  the  Smart  Board  presentation  and  learning  format  with  or  without 
accompanying  video  simulations.  One  focus  of  the  case-based  systems  knowledge  teaching 
units  would  be  to  illustrate  system  interdependencies  —  and  the  need  for  implicational 
thinking  and  pre-planning  to  deal  with  these.  Given  the  high  costs  associated  with  linking 
the  dynamic  systems  portrayed  in  the  Smart  Board  modules,  this  is  the  most  effective  and 
cost-efficient  approach  to  teaching  causal  inter-linkage  between  aircraft  systems.  In  order  to 
overcome  a  main  obstacle  to  case-based  teaching,  the  lack  of  suitably  developed  cases,  it  is 
recommended  that  training  partners  in  the  world-wide  CC-130  community  be  found  that 
would  agree  to  share  their  cases. 
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3.  Leverage  the  Smart  Board  software  pieces.  Deploy  the  Smart  Boards  at  both  the  OTU  and 
OJT  within  the  context  of  both  problem-based  learning  and  self-study.  Combine  them  with  a 
case-based  method  of  instruction,  and  intelligent  tutoring  for  self-directed,  continuous 
performance  improvement.  Build  the  tutors  to  provide  feedback  and  develop  pilots’  abilities 
for  objective,  independent  self-evaluation  of  the  state  of  their  systems’  knowledge. 

4.  Teach  the  AIP  model  as  a  conceptual  approach,  at  an  early  stage  and  throughout  training  so 
that  it  becomes  habitual. 

5.  Conduct  formal  evaluation  studies  to  document  the  efficacy  of  these  value-added 
improvements. 

Recommendations  for  Implementation 

Systems  knowledge  training  will  be  multi-stage,  involving  both  426  and  Squadron-based  training 

and  instruction.  It  should  incorporate: 

•  formal  training  opportunities; 

•  strong  conceptual  mental  model  building; 

•  adequate  practice  opportunities  for  the  development  of  skill  (including  dynamic  decision 
making  with  respect  to  systems  wherever  possible  in  simulations,  part-task  trainers,  etc.); 

•  case-based  learning; 

•  exposure  to  experts; 

•  self-study  training;  and 

•  feedback,  reinforcement,  and  evaluation  for  the  student. 

Details  of  the  implementation  should  be  worked  out  with  8  Wing  and  426  Squadron.  The 

following  tasks  will  need  to  be  considered  to  complete  the  full  implementation: 

•  determining  the  needed  staffing  complement  at  426; 

•  determining  the  staffing  and  support  needed  at  Squadron; 

•  an  organisational  analysis  of  the  co-ordination  needed  between  426  trainers  and  Standards 
Officers  at  Squadron; 

•  a  training  needs  analysis  that  stipulates  the  skills  demonstrable  at  proficiency  for  each  aircraft 
system; 

•  a  training  needs  analysis  that  views  each  system  in  the  context  of  its  operational  environment, 
for  tailored  systems  training  at  Squadron; 

•  an  evaluation  of  how  much  detail  needs  to  be  imparted  at  426  vice  Squadrons; 

•  determining  the  needed  level  of  computer  literacy  for  students  and  trainers,  and  delivering 
this  training; 

•  training  the  trainers/Squadron  officers  in  recent  advances  in  case-based  teaching  theory  and 
their  roles  as  facilitators; 

•  determining  how  to  make  self-study  packages  accessible  to  students  (e.g.,  via  extended  426 
hours,  inter/intra  net  access,  etc.); 

•  completion  of  all  deliverables  (courseware,  caseware,  etc.);  and 

•  a  formal  efficacy  study  of  the  training. 
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CONCLUSIONS 


Despite  downsizing,  and  corporate  raiding  of  its  pilots,  1  CAD,  through  its  OTU  and  OJT, 
continues  to  deliver  a  quality  training  product  in  the  area  of  systems  knowledge  acquisition,  and 
continues  to  provide  innovative  approaches  (e.g.  Smart  Board  Systems)  to  training.  However, 
when  compared  to  other  fleets,  and  reviewed  in  terms  of  the  education  and  training  literature^ 
systems  knowledge  training  is  in  need  of  modernisation  in  terms  of  both  effectiveness  and 
efficiency  of  training  philosophy,  materials,  methods,  and  evaluation. 

Modernisation  of  systems  knowledge  training  is  especially  important  in  the  current  context  of 
lowered  experience  levels,  as  a  compensatory  defence  against  latent  failures  that  otherwise  would 
be  caused  by  insufficiently  prepared  pilots  or  inadequately  rigorous  training  regimes  within  the 
line  management  system.  Modernisation  of  systems  knowledge  training  methods  is  also  timely 
in  light  of: 

•  training  advances  made  for  training  semi-glass  and  cockpit  flightdecks; 

•  the  impending  conversion  of  the  CC-130  fleet  to  the  semi-glass  cockpit;  and 

•  the  identification  (both  in  the  aviation  community  in  general,  and  within  the  CF  CC- 
130  community  in  particular)  of  training  for  systems  knowledge  complexity  as  a 
major  challenge  for  aircraft  so  equipped. 

Modernised  systems  knowledge  training  will  need  to  be: 

•  multi-staged  and  progressive; 

•  distributed  more  to  OJT  for  improved  integrated  skill  development; 

•  problem-,  or  case-,  based; 

•  retooled  to  add  higher  order  cognitive  functioning  with  regard  to  systems  (e.g., 
systems  situational  awareness,  implicational  thinking  regarding  systems  problems, 
and  planning  for  system  and  inter-system  contingencies); 

•  career-long  via  self-study  opportunities,  which  include  the  development  of  self- 
evaluation  skills  based  on  meta-knowledge  (i.e.,  appraising  what  you  know, 
realistically  knowing  what  you  don’t  know,  and  identifying  what  you  need  to  know  ); 

•  based  on  efficient  training  tools  such  as  interactive  multi-media  simulations 
incorporating  performance  feedback;  and 

•  evaluated  in  the  formal  terms  of  an  efficacy  study. 

Fleet-wide  efficacy  of  the  training  over  time,  as  well  as  long-term  trends  in  absolute  levels  of 
systems  knowledge  fleet-wide,  should  be  monitored  via  a  no-jeopardy  database  of  systems 
knowledge  evaluations  collected  from  several  sources: 

•  from  systems  knowledge  performance  evaluated  on  exit  from  the  Conversion  and  left 
seat  upgrade  course; 


•  from  systems  knowledge  performance  in  appropriately  designed,  event-based 
scenarios  given  during  continuation  training  simulator  sessions  (i.e.,  scenarios  that 
incorporate  systems  knowledge  demands);  and 

•  in  terms  of  reported  incidents  and  accidents  involving  poor  systems  knowledge. 

Implementation  of  the  systems  knowledge  training  proposed  in  this  Volume  will  require  an 
investment  in: 

•  the  Smart  Boards  System  as  the  comer-stone  training  tool; 

•  leveraging  those  CDs  to  even  greater  educational  benefit  (e.g.,  by  adding  the  AIP  — 
Awareness,  Implications,  and  Pre-planning  —  heuristic,  self-study,  generation  of  a 
library  of  training/simulation  cases,  use  of  simulation  in  concert  with  video  re¬ 
enactments,  development  of  intelligent  tutors  for  self-study,  etc.); 

•  hardware  and  software  infrastructure  improvements  at  OJT; 

•  training  both  trainers  (OTU  and  OJT)  and  trainees  in  the  methods  of  case-based 
instruction; 

•  additional  training  personnel  assigned  to  the  OTU  to  oversee  the  improvements, 
modernisation,  and  direction  of  the  OJT  portion  of  the  training;  and 

•  general  IT  and  computer  training  support. 

Key  benefits  of  the  recommended  systems  knowledge  training  modernisation  are: 

•  increased  safety; 

•  increased  operational  efficiency;  and 

•  a  potentially  saleable  training  package  (i.e.,  philosophy,  structure,  tools,  evaluation, 
etc.)  to  the  C-130  community  world- wide,  and  to  the  CF  fleet-wide. 
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RESOURCES 


The  following  resources  have  been  assembled  from  a  variety  of  sources.  Most  are  brief 
descriptions  culled  from  the  Proceeding  of  the  Society  for  Applied  Learning  Technologies’ 
(SALT)  annual  conference  entitled.  Interactive  Multimedia  ‘96,  which  was  held  from  August  21 
to  23,  1996  in  Washington,  DC  and  attended  by  the  author.  A  copy  of  these  proceedings  is  held 
at  DCIEM. 


Lifelong  Learning  Resources 

Contact:  The  Canadian  Alliance  for  Lifelong  Learning  (CALL) 

Suite  400,  20  Bay  Street, 

Toronto,  ON 
M5J  2W1 

Telephone:  (416)325-4211  . . _ 

Fax:  (416)325-4675 

e-mail:  CALL@gsro.carlton.ca 
Web:  http://gcll.carlton.ca 

CALL  is  a  non-profit  organisation  committed  to  developing  an  action  plan  for  lifelong  learning 
in  Canada  and  around  the  globe.  It  is  associated  with  similar  organisations  around  the  world 
through  its  membership  in  the  World  Initiative  in  Lifelong  Learning,  based  in  Brussels. 

One  of  the  major  foci  of  this  alliance  is  on  practical  issues  involved  in  the  implementation  of 
learner-centred  environments,  methodologies,  technologies,  and  social  strategies.  They  keep  a 
case-study  catalogue  of  actual  examples  of  workplace  models  of  continuous  learning,  geared  to 
the  application  of  the  newest  technology  or  the  newest  learning  models,  such  as  learning 
organisations,  self-directed  learning,  learning  via  the  Internet,  the  electronic  classroom,  distance 
education,  etc. 

Because  the  lifelong  learning  philosophy  and  methods  such  as  problem-based  learning  are  now 
being  applied  down  to  the  kindergarten  level,  they  span  all  age  ranges  in  education.  However, 
there  is  a  particular  focus  within  this  span  on  adult  education  in  general  (e.g.,  within  human 
resources  development  on  productivity  improvement  strategies  based  on  continuous  learning  and 
career  planning)  and  on  workplace  training  in  particular  (e.g.,  creating  and  sustaining  learning 
environments  within  govemment/industry  environments;  impact  of  new  technologies;  electronic 
classroomsTetc . ) . 
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Self-Directed  Learning  Resources 


Title:  The  McMaster  Medical  Education  Philosophy 

Contact:  Health  Science  Centre 

Faculty  of  Health  Sciences 
McMaster  University 
1200  Main  Street  West 
Hamilton,  ON 
L8N  3Z5 

Telephone:  (905)  525-9140 

Since  its  inception  in  1969,  McMaster  University’s  Health  Science  Centre  has  pioneered  radical 
innovations  in  medical  education.  With  nearly  30  years  experience  in  the  student-centred, 
problem-based,  self-  and  group-directed  learning  philosophy,  it  stands  as  an  excellent  Canadian 
resource  for  the  introduction  of  effective  new  methods  in  the  lifelong  learning  field.  Widely 
copied  both  within  medicine  (e.g.,  Harvard  Medical  School)  and  without  (all  other  McMaster 
faculties),  it  is  a  respected  source  of  information.  McMaster  holds  a  week-long,  biannual 
visitor’s  workshop  which  serves  as  a  briefing  programme  to  its  methods. 


Conference  on  Interactive  Multimedia  ’96 

This  conference  was  sponsored  by  the  Society  for  Applied  Learning  Technologies  (SALT). 
Section  headings  reflect  particular  themes.  The  following  presentations  are  noteworthy. 


Performance  Support  System  Rescues 

Title:  Use  of  the  Internet  to  Assist  in  Creating  CBT,  Performance  Support  And 

Interactive  Learning  Programs 

Author:  John  Kellum,  VP  and  GM  of  Multimedia  Tools,  Asymetrix  Corp. 

How  to  create  quick,  easy,  cost-effective  CBT,  PSS,  and  interactive  learning  applications. 
Supplied  internal  education  to  Boeing. 

Title:  How  to  Design  and  Build  User-Centred  Integrated  Training  and  Performance 

Support  System 

Authors:  Janis  A.  Morariu,  Peak  Potential,  Inc.  and  Eric  J.  Malecki,  M  Multimedia,  Ltd. 

Describes  their  Integrated  Training  and  Performance  Support  (FTPS)  Design  Methodology, 
which  includes  5  types  of  goals/requirements  and  proceeds  through  five  stages.  Case  study 
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example  is  the  FAA’s  Training  and  Evaluation  Support  System  (TESS),  developed  according  to 
the  methodology,  as  the  FAA’s  prototype  model  for  determining  training  cost-benefits  and 
Return  on  Investment  (ROI). 


Title:  Performance  WebSM  Case  Studies:  How  Training  Organizations  Are  Using  the 

World  Wide  Web 

Author:  Michael  Gallagher,  Director,  Sales  and  Marketing,  AERA,  Inc. 

Application  of  browsers  to  presentation  of  technical  documentation  and  training  via  intranets. 
Case  studies  include  NASA  Goddard. 


Special  Design  Resources 

Title:  Incorporating  Cognitive  Strategies  and  Metacognition  into  Interactive  CBT 

Author:  Anne-Marie  Armstrong,  Instructional  Designer,  AERA 

Provides  a  guide  to  designing  screens  that  meet  cognitive  learning  objectives,  as  well  as 
evaluation  of  such  objectives. 


Title:  Work  Breakdown  Structures  (WBS):  An  Automated  Systems  Approach  To 

Planning  Technical  Training 

Author:  Darrell  Tatro,  Program  Coordinator,  AEGIS  Training  Center,  and  Marilyn,  B. 

Newman,  Training  Specialist,  AEGIS  Training  Center 

Aid  to  planning  and  implementing  large-scale  training  programs;  can  be  used  as  a  methodology 
for  planning  overall  training  strategy,  and  multimedia  training/tools  within  this  strategy. 


World  Wide  Web  Training  Resources 

Title:  Internet  Access  for  Beginners  (CD-ROM) 

Authors:  Mary  E.  Sand,  Program  Manager,  Technical  Training,  FAA,  and  Nicholas 

Chandler,  Director  of  Training  Systems,  New  Media  Division,  SAIC 

A  CBI  course  that  teaches  and  uses  the  skills  needed  to  distribute  and  maintain  text-based  and 
other  media-based  teaching/leaming  products  over  the  Internet  (including  CBI  multi-media 
materials).  The  course  teaches  beginning  Internet  skills  so  as  to  avoid  user  frustration,  time 
wastage,  etc.,  while  also  serving  as  an  introduction  to  CBT-based  education  and  training 
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Title: 


Training  on  the  World  Wide  Web:  Alternatives  in  Design  and  Development 


Authors:  Ann  Barron,  Associate  Professor,  Instructional  Technology,  University  of  South 

Florida,  Brendan  Tompkins,  Programmer,  and  Kord  Kutchines,  Programmer, 
Analysis  and  Technology,  Inc. 

Presents  new  authoring  tools,  programming  languages  for  interactive  courseware  creation, 
including  HTML,  Java,  Shockwave,  IconAuthor  and  PERL.  Design  guidelines  are  enumerated 
as  well. 


Title:  How  to  Create  Your  Own  Internet-Based  Training  Course 

Author:  Pardner  Wynn,  President,  Stanford  Testing  Systems,  Inc. 

Demonstration  of  how  to  create  a  complete  Internet-Based  Training  (IBT)  course  using 
IBTauthor,  an  off-the-shelf  authoring  tool. 


Title:  US  Department  of  Defense  On-Line  Internet  Journal  for  Training  and 

Performance 

Authors:  Leslye  O.  Fuller,  Instructional  Technologist,  Defense  Business  Management 

University,  Bart  Morrison,  Senior  Analyst,  Analysis  and  Technology,  Inc.,  and 
Thomas  J.  Edwards,  Combined  Arms  Support  Command,  US  Army 

On-Line  journal  mounted  to  leverage  the  shared  experience  of  US  forces’  application  of 
technology  to  education  and  training.  Idea  exchange  via  usegroups,  topical  forums, 
downloadable  journal  articles,  demonstrations  of  new  technology,  etc.  Topic  areas  include 
human  factors,  cognition,  CBT,  intelligent  tutors,  simulation,  distance  education,  adult 
education,  multimedia,  training  devices,  etc. 


Title:  Internet  Treasures  for  Trainers 

Author:  Jodi  Bollaert,  Instructional  Technologist,  Ford  Motor  Company 

Listing  of  web  sites,  gophers,  and  newsgroups  for  training  specialists. 

Title:  Air  Force  Distance  Training 

Author:  Thomas  E.  Wolfe,  Chief  of  Distance  Learning  Instructional  Design  Branch,  Air 

Force  Distance  Learning  Office 
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Covers  the  expansive  growth  of  distance  learning  in  the  USAF  with  CBI,  interactive  courseware, 
and  interactive  multimedia.  Exhaustive  coverage  suitable  to  an  audience  from  beginners  to 
practitioners,  dealing  with  intranets  to  MBONE. 


Multimedia  Development  Resources 

Title:  Storyboarding  That  Really  Supports  Multimedia  Development 

Authors:  Gary  E.  Davis,  Multimedia  Development  Specialist;  Lucent  Technologies,  and 

Robert  C.  Fratini,  Senior  Advanced  Media  Consultant,  AT&T  Network  Systems; 
Lucent  Technologies 

Lesson  planning  for  case  study  presentation  via  multimedia  can  be  done  easily  by  storyboarding. 
This  paper  describes  the  organisational  factors  needed  to  support  successful  storyboarding,  as 
well  the  Multimedia  Information  Modeler  to  be  used  by  the  SME  in  lesson  planning*  for 
multimedia. 


Title:  The  Power  of  Interactive  Video  for  FAA  Course  Development  and  Delivery: 

Lessons  Learned 

Author:  Phyllis  Peters  Marson,  FAA 

Paper  on  interactive  video  training  in  the  FAA.  Only  marginally  relevant,  but  included  here 
should  the  interactive  teleconferencing  of  course  delivery  ever  be  considered. 


Title:  Interactive  Multimedia  Learning  Systems 

Author:  Michael  Dulworth,  Insync  Corp. 

A  good  primer  of  multimedia  learning  systems,  including  how  learning  theory  applies,  the 
development  process,  and  authoring  systems.  Includes  a  brief  description  of  the  four  levels  of 
multimedia  learning:  from  customised  linear  presentation,  to  instructor-led/non-linear 
presentation,  to  facilitator-led  training,  to  self-paced  training. 


Title.  Rapid  Scripting  —  A  Non-Traditional  Approach  to  Storyboard  Development 

Author:  Mary  F.  Bratton- Jeffery,  US  Navy 

Current  trends  in  authoring  packages.  Rapid  scripting  was  developed  to  go  beyond  paper  and 
pencil  development  tools  and  produce  synergistic  effects  among  interactive  courseware 
production  team  members. 
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Training  and  Development  Resources 


Title:  The  After  Action  Review  Process:  A  Heuristic  Computer  Assisted  Tool  For 

Training 

Author:  Robert  C.  Hughes,  Program  Manager,  Cubic  Applications 


After  Action  Review  (AAR)  is  a  computer-assisted  process  designed  to  provide  feedback  after 
computer-assisted  training,  and  to  evaluate  decision-making,  courses  of  actions  selected, 
performance  against  objectives  and  team  interactions.  The  process  promotes  metacognitive 
development  and  awareness  of  the  epistemology  of  what  people  are  actually  doing,  and  is  highly 
compatible  with  the  systems  approach  to  training  and  education  we  have  proposed.  A  meiotic 
teaching  approach,  it  is  based  on  Bloom’s  taxonomy  of  educational  objectives  and  the  metaphor 
of  "candling”,  that  is,  observing  without  disturbing  the  group  and  learning  process.  R.  Hughes  is 
a  former  professor  in  the  US  Air  Force  Academy. 


Title:  Interactive  Multimedia:  Cost  Benefit  Analysis  Issues 

Author:  John  J.  Hirschbuhl,  Assistant  to  the  President,  Information  Services,  The 

University  of  Akron 

Illustration  by  case  studies  of  cost-benefit  analysis  and  positive  ROI  for  networked  multimedia  in 
corporate  environments.  Surprising  results  have  been  achieved.  For  example,  the  US  Institute 
for  Defense  Analysis,  IBM,  and  American  Airlines  indicate  that  "computer-based  multimedia 
produced  a  savings  of  68%  over  conventional  classroom  training"  (p.  4). 


Innovative  Technologies 

Title:  Automated  Speech  Technologies:  Implications  for  Training  and  Education 

Authors:  Lyn  Gubser,  President,  American  Insights,  Inc.,  and  Thomas  H.  Held,  Director 

Multimedia,  Marketing  and  Product  Development,  Cubic  Applications,  Inc. 

Review  of  work  at  leading  laboratories  in  language  instruction  and  translation,  natural  language 
recognition  systems  and  text-to-speech  (TTS)  systems.  Describes  the  state  of  the  art  in 
Automated  Speech  Technologies  (AST).  This  is  potentially  useful  in  transcription  of  verbal 
protocols  in  full  motion  simulator  sessions  testing  systems  knowledge. 


Title:  On-Line  Dissemination  of  Expert  Knowledge  on  the  Web 
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Authors: 


Dustin  Huntington ,  EXSYS  Inc.,  and  Elias  M.  Awad,  Mclntire  School  of 
Commerce,  University  of  Virginia 


Leading  expert  systems  company  EXSYS  presents  their  approach  to  capturing  and  making 
accessible,  expert  knowledge.  The  system  asks  the  user  focused  relevant  questions  and  comes  to 
conclusions  about  the  state  if  their  knowledge  based  on  the  answers.  Users  are  allowed  to  ask 
questions  in  order  to  expose  the  knowledge  and  inference  structures  of  the  expert  knowledge 
representation.  Provides  explanation  and  recommendations.  Expert  knowledge  is  built  using 
expert  knowledge  development  tools  that  are  files  containing  decision-making  knowledge 
expressed  as  rules  or  facts. 
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